2020-07-23T12:13:04.681 INFO:root:teuthology version: 1.0.0-bb8d0b5 2020-07-23T12:13:04.687 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --block --owner scheduled_teuthology@teuthology --lock --description rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} --archive /home/teuthworker/archive/teuthology-2020-07-20_05:05:02-rgw-octopus-distro-basic-smithi/5242029 --name teuthology-2020-07-20_05:05:02-rgw-octopus-distro-basic-smithi -- /tmp/teuthology-worker.vuasxsfx.tmp 2020-07-23T12:13:04.748 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-07-23T12:13:04.835 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2020-07-20_05:05:02-rgw-octopus-distro-basic-smithi/5242029 branch: octopus description: rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: ceph-qa@ceph.io first_in_suite: false job_id: '5242029' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2020-07-20_05:05:02-rgw-octopus-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: octopus ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac rgw: compression type: random frontend: civetweb rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: octopus sha1: bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac owner: scheduled_teuthology@teuthology priority: 1000 repo: git://git.ceph.com/ceph.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac sleep_before_teardown: 0 suite: rgw suite_branch: octopus suite_path: /home/teuthworker/src/git.ceph.com_ceph_octopus/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph.git suite_sha1: bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master tube: smithi verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.20760 2020-07-23T12:13:04.835 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_octopus/qa; will attempt to use it 2020-07-23T12:13:04.836 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_octopus/qa/tasks 2020-07-23T12:13:04.837 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-07-23T12:13:04.843 INFO:teuthology.task.internal:Checking packages... 2020-07-23T12:13:04.882 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac' 2020-07-23T12:13:04.883 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-07-23T12:13:04.884 INFO:teuthology.packaging:ref: None 2020-07-23T12:13:04.884 INFO:teuthology.packaging:tag: None 2020-07-23T12:13:04.885 INFO:teuthology.packaging:branch: octopus 2020-07-23T12:13:04.885 INFO:teuthology.packaging:sha1: bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac 2020-07-23T12:13:04.886 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=octopus 2020-07-23T12:13:05.282 INFO:teuthology.task.internal:Found packages for ceph version 15.2.4-391-g7afebc4-1bionic 2020-07-23T12:13:05.282 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-07-23T12:13:05.294 INFO:teuthology.task.internal:no buildpackages task found 2020-07-23T12:13:05.295 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-07-23T12:13:05.322 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-07-23T12:13:05.323 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-07-23T12:13:05.398 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_teuthology@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/teuthology-2020-07-20_05:05:02-rgw-octopus-distro-basic-smithi/5242029'} 2020-07-23T12:13:05.429 DEBUG:teuthology.lock.ops:locked ubuntu@smithi137.front.sepia.ceph.com, ubuntu@smithi133.front.sepia.ceph.com 2020-07-23T12:13:05.542 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd2e54a7510>, signals=[15]) 2020-07-23T12:13:05.798 INFO:teuthology.provision.fog.smithi133:Scheduling deploy of ubuntu 18.04 2020-07-23T12:13:05.835 INFO:teuthology.provision.fog.smithi137:Scheduling deploy of ubuntu 18.04 2020-07-23T12:13:06.286 INFO:teuthology.orchestra.console:Power off smithi133 2020-07-23T12:13:06.288 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-07-23T12:13:06.354 INFO:teuthology.orchestra.console:Power off smithi137 2020-07-23T12:13:06.355 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-07-23T12:13:06.374 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-23T12:13:06.404 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-23T12:13:10.422 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-23T12:13:10.444 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-23T12:13:14.547 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-23T12:13:14.564 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-23T12:13:14.766 INFO:teuthology.orchestra.console:Power off for smithi133 completed 2020-07-23T12:13:14.867 INFO:teuthology.orchestra.console:Power on smithi133 2020-07-23T12:13:14.868 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-07-23T12:13:14.892 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-23T12:13:18.683 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-23T12:13:18.901 INFO:teuthology.orchestra.console:Power off for smithi137 completed 2020-07-23T12:13:18.926 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-23T12:13:19.013 INFO:teuthology.orchestra.console:Power on smithi137 2020-07-23T12:13:19.016 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-07-23T12:13:19.047 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-23T12:13:23.043 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-23T12:13:23.440 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-23T12:13:23.662 INFO:teuthology.orchestra.console:Power on for smithi133 completed 2020-07-23T12:13:23.763 INFO:teuthology.provision.fog.smithi133:Waiting for deploy to finish 2020-07-23T12:13:27.554 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-23T12:13:27.786 INFO:teuthology.orchestra.console:Power on for smithi137 completed 2020-07-23T12:13:27.888 INFO:teuthology.provision.fog.smithi137:Waiting for deploy to finish 2020-07-23T12:16:10.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:16:19.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:16:28.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:16:29.266 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:16:37.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:16:38.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:16:46.516 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:16:47.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:16:55.604 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:16:56.434 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:17:04.694 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:17:05.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:17:13.778 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:17:14.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:17:23.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:18:19.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:18:19.933 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-07-23T12:18:19.941 INFO:teuthology.orchestra.run.smithi133:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-07-23T12:18:20.295 INFO:teuthology.provision.fog.smithi133:Node is ready 2020-07-23T12:18:20.296 INFO:teuthology.orchestra.run.smithi133:> hostname 2020-07-23T12:18:20.345 INFO:teuthology.orchestra.run.smithi133.stdout:smithi133.front.sepia.ceph.com 2020-07-23T12:18:20.350 INFO:teuthology.orchestra.run.smithi133:> grep smithi133.front.sepia.ceph.com /etc/hosts 2020-07-23T12:18:20.409 INFO:teuthology.orchestra.run.smithi133.stdout:172.21.15.133 smithi133.front.sepia.ceph.com smithi133 2020-07-23T12:18:20.411 INFO:teuthology.orchestra.run.smithi133:> sudo hostname smithi133 2020-07-23T12:18:20.518 INFO:teuthology.orchestra.run.smithi133:> sudo sed -i -e 's/smithi133.front.sepia.ceph.com/smithi133/g' /etc/hosts 2020-07-23T12:18:20.553 INFO:teuthology.orchestra.run.smithi133:> sudo sed -i -e 's/172.21.15.133/172.21.15.133/g' /etc/hosts 2020-07-23T12:18:20.609 INFO:teuthology.orchestra.run.smithi133:> sudo sed -i -e 's/smithi133.front.sepia.ceph.com/smithi133/g' /etc/hostname 2020-07-23T12:18:20.664 INFO:teuthology.orchestra.run.smithi133:> sudo hostname smithi133 2020-07-23T12:18:20.720 INFO:teuthology.provision.fog.smithi133:Deploy complete! 2020-07-23T12:18:20.721 DEBUG:teuthology.parallel:result is None 2020-07-23T12:18:29.611 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:18:29.902 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-07-23T12:18:29.903 INFO:teuthology.orchestra.run.smithi137:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-07-23T12:18:30.287 INFO:teuthology.provision.fog.smithi137:Node is ready 2020-07-23T12:18:30.288 INFO:teuthology.orchestra.run.smithi137:> hostname 2020-07-23T12:18:30.336 INFO:teuthology.orchestra.run.smithi137.stdout:smithi137.front.sepia.ceph.com 2020-07-23T12:18:30.337 INFO:teuthology.orchestra.run.smithi137:> grep smithi137.front.sepia.ceph.com /etc/hosts 2020-07-23T12:18:30.386 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.15.137 smithi137.front.sepia.ceph.com smithi137 2020-07-23T12:18:30.387 INFO:teuthology.orchestra.run.smithi137:> sudo hostname smithi137 2020-07-23T12:18:30.528 INFO:teuthology.orchestra.run.smithi137:> sudo sed -i -e 's/smithi137.front.sepia.ceph.com/smithi137/g' /etc/hosts 2020-07-23T12:18:30.546 INFO:teuthology.orchestra.run.smithi137:> sudo sed -i -e 's/172.21.15.137/172.21.15.137/g' /etc/hosts 2020-07-23T12:18:30.602 INFO:teuthology.orchestra.run.smithi137:> sudo sed -i -e 's/smithi137.front.sepia.ceph.com/smithi137/g' /etc/hostname 2020-07-23T12:18:30.663 INFO:teuthology.orchestra.run.smithi137:> sudo hostname smithi137 2020-07-23T12:18:30.741 INFO:teuthology.provision.fog.smithi137:Deploy complete! 2020-07-23T12:18:30.742 DEBUG:teuthology.parallel:result is None 2020-07-23T12:18:30.962 INFO:teuthology.lock.ops:Checking smithi137.front.sepia.ceph.com 2020-07-23T12:18:30.964 INFO:teuthology.lock.ops:New key found. Updating... 2020-07-23T12:18:30.997 INFO:teuthology.lock.ops:Checking smithi133.front.sepia.ceph.com 2020-07-23T12:18:30.998 INFO:teuthology.lock.ops:New key found. Updating... 2020-07-23T12:18:31.044 INFO:teuthology.lock.ops:Updating [smithi137.front.sepia.ceph.com]: set os type and version on server 2020-07-23T12:18:31.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-23T12:18:31.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:18:31.172 INFO:teuthology.orchestra.run.smithi137:> true 2020-07-23T12:18:31.363 INFO:teuthology.orchestra.run.smithi137:> uname -m 2020-07-23T12:18:31.420 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2020-07-23T12:18:31.421 INFO:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2020-07-23T12:18:31.470 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="Ubuntu" 2020-07-23T12:18:31.471 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-07-23T12:18:31.471 INFO:teuthology.orchestra.run.smithi137.stdout:ID=ubuntu 2020-07-23T12:18:31.471 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE=debian 2020-07-23T12:18:31.472 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-07-23T12:18:31.472 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="18.04" 2020-07-23T12:18:31.472 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://www.ubuntu.com/" 2020-07-23T12:18:31.473 INFO:teuthology.orchestra.run.smithi137.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-07-23T12:18:31.473 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-07-23T12:18:31.473 INFO:teuthology.orchestra.run.smithi137.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-07-23T12:18:31.479 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_CODENAME=bionic 2020-07-23T12:18:31.479 INFO:teuthology.orchestra.run.smithi137.stdout:UBUNTU_CODENAME=bionic 2020-07-23T12:18:31.484 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2020-07-23T12:18:31.506 INFO:teuthology.lock.ops:Updating [smithi133.front.sepia.ceph.com]: set os type and version on server 2020-07-23T12:18:31.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-23T12:18:31.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:18:31.606 INFO:teuthology.orchestra.run.smithi133:> true 2020-07-23T12:18:31.868 INFO:teuthology.orchestra.run.smithi133:> uname -m 2020-07-23T12:18:31.926 INFO:teuthology.orchestra.run.smithi133.stdout:x86_64 2020-07-23T12:18:31.927 INFO:teuthology.orchestra.run.smithi133:> cat /etc/os-release 2020-07-23T12:18:31.979 INFO:teuthology.orchestra.run.smithi133.stdout:NAME="Ubuntu" 2020-07-23T12:18:31.979 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-07-23T12:18:31.980 INFO:teuthology.orchestra.run.smithi133.stdout:ID=ubuntu 2020-07-23T12:18:31.980 INFO:teuthology.orchestra.run.smithi133.stdout:ID_LIKE=debian 2020-07-23T12:18:31.980 INFO:teuthology.orchestra.run.smithi133.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-07-23T12:18:31.980 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_ID="18.04" 2020-07-23T12:18:31.981 INFO:teuthology.orchestra.run.smithi133.stdout:HOME_URL="https://www.ubuntu.com/" 2020-07-23T12:18:31.981 INFO:teuthology.orchestra.run.smithi133.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-07-23T12:18:31.981 INFO:teuthology.orchestra.run.smithi133.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-07-23T12:18:31.982 INFO:teuthology.orchestra.run.smithi133.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-07-23T12:18:31.982 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_CODENAME=bionic 2020-07-23T12:18:31.983 INFO:teuthology.orchestra.run.smithi133.stdout:UBUNTU_CODENAME=bionic 2020-07-23T12:18:31.984 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2020-07-23T12:18:32.011 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-07-23T12:18:32.069 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi133.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDTunE7kpOkFCjYvZc7sF3Pj+bcPT2jzsHSBbPaEA/tW/bjDZRdd8emTC4L0fxv0CG02maN33JHAxJdBoLaUZuGuGaExQ0q3xlxBxslCwAn7h+uWJdYS3ZTtHDriAcNG1tK0zAy7hkF1WTJptRrQGhEMp4R+cKZqkNnrl71yMvrTCxK+RVdWdx92U4i476ggc0dQZdKst52B9oUJ6dU4fu2Ivn5GjWiYk1Voy/TyRaG8TgVEyvpKJ38UURB1Axb/P/60/Xn9WDsWp3k25z5uw3dzdEC6z3z8i8jUpHssvD0l7CGxxRZiwkt3MoKH4XY5J7cI1Ynm9QjI2YcQqqh6P8z smithi137.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDujhwHQQYYd0i9/NuavIxWtoPBK11AcchJKBZRpQXB49v1LqcluhHxhXvo8P17ads2mEYs5fiU6zWmywcGE9JJ2skLImeTRfEaTwuAmAMKfGZ1Lv9jX3EHAGViMTYcpkY59w462r7A+d5gt9JY/8aNlWdNdQgEY3GA7h6iZuixavroXd28UB3wzTVLOrCCTpYDMhDlvmioE63/iXbtwUE0UYhEAsnpWk7nx8kLw+nRrHs1QPlH447Vpqpb/zx0zMeJ9e/Xvu3KyHImTa8P9eV59CW6j3FnufcbWAl/NxI6lgjXfqJQO0f4PZSlM95kS94Yp90nqD2FzaxXrQpXhJvH 2020-07-23T12:18:32.070 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-07-23T12:18:32.131 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-07-23T12:18:32.197 INFO:teuthology.task.internal:Saving configuration 2020-07-23T12:18:32.225 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-07-23T12:18:32.232 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-07-23T12:18:32.250 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-07-23 12:13:05.411273', 'description': '/home/teuthworker/archive/teuthology-2020-07-20_05:05:02-rgw-octopus-distro-basic-smithi/5242029', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOHOFkRHEin/VnKIf3tOG5I4CfkYWeYrhUuCLxr7GBry', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi137.front.sepia.ceph.com'} 2020-07-23T12:18:32.278 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-07-23 12:13:05.411305', 'description': '/home/teuthworker/archive/teuthology-2020-07-20_05:05:02-rgw-octopus-distro-basic-smithi/5242029', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILAzBnyfVqEfZrUCbp8vBz2fauYwXdLqJOfzsTf9pJ4M', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi133.front.sepia.ceph.com'} 2020-07-23T12:18:32.278 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-07-23T12:18:32.289 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2020-07-23T12:18:32.290 INFO:teuthology.task.internal:roles: ubuntu@smithi133.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2020-07-23T12:18:32.290 INFO:teuthology.run_tasks:Running task console_log... 2020-07-23T12:18:32.420 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd2e53dc158>, signals=[15]) 2020-07-23T12:18:32.433 INFO:teuthology.run_tasks:Running task internal.connect... 2020-07-23T12:18:32.497 INFO:teuthology.task.internal:Opening connections... 2020-07-23T12:18:32.498 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2020-07-23T12:18:32.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:18:32.625 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi133.front.sepia.ceph.com 2020-07-23T12:18:32.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:18:32.758 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-07-23T12:18:32.771 INFO:teuthology.orchestra.run.smithi137:> uname -m 2020-07-23T12:18:32.823 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2020-07-23T12:18:32.824 INFO:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2020-07-23T12:18:32.872 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="Ubuntu" 2020-07-23T12:18:32.872 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-07-23T12:18:32.872 INFO:teuthology.orchestra.run.smithi137.stdout:ID=ubuntu 2020-07-23T12:18:32.873 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE=debian 2020-07-23T12:18:32.873 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-07-23T12:18:32.873 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="18.04" 2020-07-23T12:18:32.873 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://www.ubuntu.com/" 2020-07-23T12:18:32.873 INFO:teuthology.orchestra.run.smithi137.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-07-23T12:18:32.874 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-07-23T12:18:32.874 INFO:teuthology.orchestra.run.smithi137.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-07-23T12:18:32.874 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_CODENAME=bionic 2020-07-23T12:18:32.874 INFO:teuthology.orchestra.run.smithi137.stdout:UBUNTU_CODENAME=bionic 2020-07-23T12:18:32.875 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2020-07-23T12:18:32.900 INFO:teuthology.orchestra.run.smithi133:> uname -m 2020-07-23T12:18:32.963 INFO:teuthology.orchestra.run.smithi133.stdout:x86_64 2020-07-23T12:18:32.963 INFO:teuthology.orchestra.run.smithi133:> cat /etc/os-release 2020-07-23T12:18:33.004 INFO:teuthology.orchestra.run.smithi133.stdout:NAME="Ubuntu" 2020-07-23T12:18:33.004 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-07-23T12:18:33.004 INFO:teuthology.orchestra.run.smithi133.stdout:ID=ubuntu 2020-07-23T12:18:33.004 INFO:teuthology.orchestra.run.smithi133.stdout:ID_LIKE=debian 2020-07-23T12:18:33.005 INFO:teuthology.orchestra.run.smithi133.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-07-23T12:18:33.005 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_ID="18.04" 2020-07-23T12:18:33.005 INFO:teuthology.orchestra.run.smithi133.stdout:HOME_URL="https://www.ubuntu.com/" 2020-07-23T12:18:33.005 INFO:teuthology.orchestra.run.smithi133.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-07-23T12:18:33.005 INFO:teuthology.orchestra.run.smithi133.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-07-23T12:18:33.006 INFO:teuthology.orchestra.run.smithi133.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-07-23T12:18:33.006 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_CODENAME=bionic 2020-07-23T12:18:33.006 INFO:teuthology.orchestra.run.smithi133.stdout:UBUNTU_CODENAME=bionic 2020-07-23T12:18:33.006 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2020-07-23T12:18:33.030 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-07-23T12:18:33.083 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-07-23T12:18:33.102 INFO:teuthology.task.internal:Checking for old test directory... 2020-07-23T12:18:33.103 INFO:teuthology.orchestra.run.smithi133:> test '!' -e /home/ubuntu/cephtest 2020-07-23T12:18:33.107 INFO:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2020-07-23T12:18:33.116 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-07-23T12:18:33.130 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-07-23T12:18:33.131 INFO:teuthology.orchestra.run.smithi133:> test -z $(ls -A /var/lib/ceph) 2020-07-23T12:18:33.159 INFO:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2020-07-23T12:18:33.185 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-07-23T12:18:33.239 INFO:teuthology.run_tasks:Running task kernel... 2020-07-23T12:18:33.264 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2020-07-23T12:18:33.265 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-07-23T12:18:33.265 INFO:teuthology.orchestra.run.smithi137:> uname -r 2020-07-23T12:18:33.280 INFO:teuthology.orchestra.run.smithi137.stdout:4.15.0-111-generic 2020-07-23T12:18:33.280 INFO:teuthology.task.kernel:Running kernel on smithi137: 4.15.0-111-generic 2020-07-23T12:18:33.281 INFO:teuthology.orchestra.run.smithi137:> sudo apt-get clean 2020-07-23T12:18:33.407 INFO:teuthology.orchestra.run.smithi137:> sudo apt-get update 2020-07-23T12:18:33.715 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-07-23T12:18:33.716 INFO:teuthology.orchestra.run.smithi137.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-07-23T12:18:33.754 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-07-23T12:18:34.024 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-07-23T12:18:34.105 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [510 kB] 2020-07-23T12:18:34.235 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [719 kB] 2020-07-23T12:18:34.421 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [796 kB] 2020-07-23T12:18:34.518 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,023 kB] 2020-07-23T12:18:34.521 INFO:teuthology.orchestra.run.smithi137.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [250 kB] 2020-07-23T12:18:34.547 INFO:teuthology.orchestra.run.smithi137.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [73.5 kB] 2020-07-23T12:18:34.555 INFO:teuthology.orchestra.run.smithi137.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [16.1 kB] 2020-07-23T12:18:34.557 INFO:teuthology.orchestra.run.smithi137.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [625 kB] 2020-07-23T12:18:34.599 INFO:teuthology.orchestra.run.smithi137.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [680 kB] 2020-07-23T12:18:34.609 INFO:teuthology.orchestra.run.smithi137.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [344 kB] 2020-07-23T12:18:34.628 INFO:teuthology.orchestra.run.smithi137.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [82.8 kB] 2020-07-23T12:18:34.630 INFO:teuthology.orchestra.run.smithi137.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [226 kB] 2020-07-23T12:18:34.633 INFO:teuthology.orchestra.run.smithi137.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [18.3 kB] 2020-07-23T12:18:34.636 INFO:teuthology.orchestra.run.smithi137.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,026 kB] 2020-07-23T12:18:34.684 INFO:teuthology.orchestra.run.smithi137.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,094 kB] 2020-07-23T12:18:34.717 INFO:teuthology.orchestra.run.smithi137.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [341 kB] 2020-07-23T12:18:35.592 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 8,076 kB in 2s (4,075 kB/s) 2020-07-23T12:18:36.966 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2020-07-23T12:18:37.004 INFO:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-07-23T12:18:37.130 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2020-07-23T12:18:37.396 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2020-07-23T12:18:37.396 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2020-07-23T12:18:37.464 INFO:teuthology.orchestra.run.smithi137.stderr:E: Unable to locate package linux-image-current-generic 2020-07-23T12:18:37.465 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-07-23T12:18:37.466 INFO:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-07-23T12:18:37.545 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2020-07-23T12:18:37.769 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2020-07-23T12:18:37.770 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2020-07-23T12:18:37.937 INFO:teuthology.orchestra.run.smithi137.stdout:The following additional packages will be installed: 2020-07-23T12:18:37.938 INFO:teuthology.orchestra.run.smithi137.stdout: linux-generic linux-headers-4.15.0-112 linux-headers-4.15.0-112-generic 2020-07-23T12:18:37.938 INFO:teuthology.orchestra.run.smithi137.stdout: linux-headers-generic linux-image-4.15.0-112-generic 2020-07-23T12:18:37.939 INFO:teuthology.orchestra.run.smithi137.stdout: linux-modules-4.15.0-112-generic linux-modules-extra-4.15.0-112-generic 2020-07-23T12:18:37.939 INFO:teuthology.orchestra.run.smithi137.stdout:Suggested packages: 2020-07-23T12:18:37.940 INFO:teuthology.orchestra.run.smithi137.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-07-23T12:18:37.940 INFO:teuthology.orchestra.run.smithi137.stdout:Recommended packages: 2020-07-23T12:18:37.940 INFO:teuthology.orchestra.run.smithi137.stdout: thermald 2020-07-23T12:18:37.972 INFO:teuthology.orchestra.run.smithi137.stdout:The following NEW packages will be installed: 2020-07-23T12:18:37.973 INFO:teuthology.orchestra.run.smithi137.stdout: linux-headers-4.15.0-112 linux-headers-4.15.0-112-generic 2020-07-23T12:18:37.973 INFO:teuthology.orchestra.run.smithi137.stdout: linux-image-4.15.0-112-generic linux-modules-4.15.0-112-generic 2020-07-23T12:18:37.973 INFO:teuthology.orchestra.run.smithi137.stdout: linux-modules-extra-4.15.0-112-generic 2020-07-23T12:18:37.975 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages will be upgraded: 2020-07-23T12:18:37.975 INFO:teuthology.orchestra.run.smithi137.stdout: linux-generic linux-headers-generic linux-image-generic 2020-07-23T12:18:38.153 INFO:teuthology.orchestra.run.smithi137.stdout:3 upgraded, 5 newly installed, 0 to remove and 14 not upgraded. 2020-07-23T12:18:38.154 INFO:teuthology.orchestra.run.smithi137.stdout:Need to get 65.9 MB of archives. 2020-07-23T12:18:38.154 INFO:teuthology.orchestra.run.smithi137.stdout:After this operation, 333 MB of additional disk space will be used. 2020-07-23T12:18:38.154 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-112-generic amd64 4.15.0-112.113 [13.1 MB] 2020-07-23T12:18:39.160 INFO:teuthology.orchestra.run.smithi137.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-112-generic amd64 4.15.0-112.113 [8,008 kB] 2020-07-23T12:18:39.350 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-112-generic amd64 4.15.0-112.113 [32.8 MB] 2020-07-23T12:18:40.300 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.112.100 [1,864 B] 2020-07-23T12:18:40.301 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.112.100 [2,616 B] 2020-07-23T12:18:40.301 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-112 all 4.15.0-112.113 [10.9 MB] 2020-07-23T12:18:40.602 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-112-generic amd64 4.15.0-112.113 [1,089 kB] 2020-07-23T12:18:40.628 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.112.100 [2,516 B] 2020-07-23T12:18:41.103 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 65.9 MB in 3s (25.0 MB/s) 2020-07-23T12:18:41.284 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-modules-4.15.0-112-generic. 2020-07-23T12:18:41.321 INFO:teuthology.orchestra.run.smithi137.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102053 files and directories currently installed.) 2020-07-23T12:18:41.324 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../0-linux-modules-4.15.0-112-generic_4.15.0-112.113_amd64.deb ... 2020-07-23T12:18:41.375 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-modules-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:18:42.955 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-image-4.15.0-112-generic. 2020-07-23T12:18:42.969 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../1-linux-image-4.15.0-112-generic_4.15.0-112.113_amd64.deb ... 2020-07-23T12:18:43.066 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-image-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:18:43.584 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-112-generic. 2020-07-23T12:18:43.603 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-112-generic_4.15.0-112.113_amd64.deb ... 2020-07-23T12:18:43.637 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-modules-extra-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:18:47.267 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../3-linux-generic_4.15.0.112.100_amd64.deb ... 2020-07-23T12:18:47.372 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-generic (4.15.0.112.100) over (4.15.0.111.99) ... 2020-07-23T12:18:48.090 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.112.100_amd64.deb ... 2020-07-23T12:18:48.210 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-image-generic (4.15.0.112.100) over (4.15.0.111.99) ... 2020-07-23T12:18:48.530 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-headers-4.15.0-112. 2020-07-23T12:18:48.540 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../5-linux-headers-4.15.0-112_4.15.0-112.113_all.deb ... 2020-07-23T12:18:48.578 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-headers-4.15.0-112 (4.15.0-112.113) ... 2020-07-23T12:18:53.022 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-headers-4.15.0-112-generic. 2020-07-23T12:18:53.042 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../6-linux-headers-4.15.0-112-generic_4.15.0-112.113_amd64.deb ... 2020-07-23T12:18:53.072 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-headers-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:18:55.350 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.112.100_amd64.deb ... 2020-07-23T12:18:55.449 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-headers-generic (4.15.0.112.100) over (4.15.0.111.99) ... 2020-07-23T12:18:55.782 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-modules-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:18:56.361 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-headers-4.15.0-112 (4.15.0-112.113) ... 2020-07-23T12:18:56.496 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-image-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:18:56.937 INFO:teuthology.orchestra.run.smithi137.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-112-generic 2020-07-23T12:18:56.937 INFO:teuthology.orchestra.run.smithi137.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-112-generic 2020-07-23T12:18:57.049 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-modules-extra-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:18:57.534 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-headers-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:18:57.661 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-image-generic (4.15.0.112.100) ... 2020-07-23T12:18:57.792 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-headers-generic (4.15.0.112.100) ... 2020-07-23T12:18:57.912 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-generic (4.15.0.112.100) ... 2020-07-23T12:18:58.038 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for linux-image-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:18:58.222 INFO:teuthology.orchestra.run.smithi137.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-07-23T12:18:58.223 INFO:teuthology.orchestra.run.smithi137.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-112-generic 2020-07-23T12:19:13.104 INFO:teuthology.orchestra.run.smithi137.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-07-23T12:19:13.105 INFO:teuthology.orchestra.run.smithi137.stdout:Sourcing file `/etc/default/grub' 2020-07-23T12:19:13.139 INFO:teuthology.orchestra.run.smithi137.stdout:Generating grub configuration file ... 2020-07-23T12:19:13.480 INFO:teuthology.orchestra.run.smithi137.stdout:Found linux image: /boot/vmlinuz-4.15.0-112-generic 2020-07-23T12:19:13.515 INFO:teuthology.orchestra.run.smithi137.stdout:Found initrd image: /boot/initrd.img-4.15.0-112-generic 2020-07-23T12:19:13.835 INFO:teuthology.orchestra.run.smithi137.stdout:Found linux image: /boot/vmlinuz-4.15.0-111-generic 2020-07-23T12:19:13.859 INFO:teuthology.orchestra.run.smithi137.stdout:Found initrd image: /boot/initrd.img-4.15.0-111-generic 2020-07-23T12:19:15.369 INFO:teuthology.orchestra.run.smithi137.stdout:done 2020-07-23T12:19:15.786 INFO:teuthology.orchestra.run.smithi137:> dpkg -s linux-image-generic 2020-07-23T12:19:15.811 INFO:teuthology.orchestra.run.smithi137.stdout:Package: linux-image-generic 2020-07-23T12:19:15.811 INFO:teuthology.orchestra.run.smithi137.stdout:Status: install ok installed 2020-07-23T12:19:15.812 INFO:teuthology.orchestra.run.smithi137.stdout:Priority: optional 2020-07-23T12:19:15.812 INFO:teuthology.orchestra.run.smithi137.stdout:Section: kernel 2020-07-23T12:19:15.812 INFO:teuthology.orchestra.run.smithi137.stdout:Installed-Size: 16 2020-07-23T12:19:15.812 INFO:teuthology.orchestra.run.smithi137.stdout:Maintainer: Ubuntu Kernel Team 2020-07-23T12:19:15.812 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: amd64 2020-07-23T12:19:15.813 INFO:teuthology.orchestra.run.smithi137.stdout:Source: linux-meta 2020-07-23T12:19:15.813 INFO:teuthology.orchestra.run.smithi137.stdout:Version: 4.15.0.112.100 2020-07-23T12:19:15.813 INFO:teuthology.orchestra.run.smithi137.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.9) 2020-07-23T12:19:15.813 INFO:teuthology.orchestra.run.smithi137.stdout:Depends: linux-image-4.15.0-112-generic, linux-modules-extra-4.15.0-112-generic, linux-firmware, intel-microcode, amd64-microcode 2020-07-23T12:19:15.813 INFO:teuthology.orchestra.run.smithi137.stdout:Recommends: thermald 2020-07-23T12:19:15.814 INFO:teuthology.orchestra.run.smithi137.stdout:Description: Generic Linux kernel image 2020-07-23T12:19:15.814 INFO:teuthology.orchestra.run.smithi137.stdout: This package will always depend on the latest generic kernel image 2020-07-23T12:19:15.814 INFO:teuthology.orchestra.run.smithi137.stdout: available. 2020-07-23T12:19:15.815 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-111-generic Expected: 4.15.0-112-generic 2020-07-23T12:19:15.815 INFO:teuthology.orchestra.run.smithi133:> uname -r 2020-07-23T12:19:15.823 INFO:teuthology.orchestra.run.smithi133.stdout:4.15.0-111-generic 2020-07-23T12:19:15.824 INFO:teuthology.task.kernel:Running kernel on smithi133: 4.15.0-111-generic 2020-07-23T12:19:15.824 INFO:teuthology.orchestra.run.smithi133:> sudo apt-get clean 2020-07-23T12:19:15.993 INFO:teuthology.orchestra.run.smithi133:> sudo apt-get update 2020-07-23T12:19:16.137 INFO:teuthology.orchestra.run.smithi133.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-07-23T12:19:16.258 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-07-23T12:19:16.342 INFO:teuthology.orchestra.run.smithi133.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-07-23T12:19:16.422 INFO:teuthology.orchestra.run.smithi133.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [796 kB] 2020-07-23T12:19:16.514 INFO:teuthology.orchestra.run.smithi133.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [510 kB] 2020-07-23T12:19:16.526 INFO:teuthology.orchestra.run.smithi133.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [250 kB] 2020-07-23T12:19:16.538 INFO:teuthology.orchestra.run.smithi133.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [73.5 kB] 2020-07-23T12:19:16.538 INFO:teuthology.orchestra.run.smithi133.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [16.1 kB] 2020-07-23T12:19:16.538 INFO:teuthology.orchestra.run.smithi133.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [680 kB] 2020-07-23T12:19:16.545 INFO:teuthology.orchestra.run.smithi133.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [625 kB] 2020-07-23T12:19:16.558 INFO:teuthology.orchestra.run.smithi133.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [226 kB] 2020-07-23T12:19:16.640 INFO:teuthology.orchestra.run.smithi133.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-07-23T12:19:16.771 INFO:teuthology.orchestra.run.smithi133.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [719 kB] 2020-07-23T12:19:17.070 INFO:teuthology.orchestra.run.smithi133.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,023 kB] 2020-07-23T12:19:17.162 INFO:teuthology.orchestra.run.smithi133.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [344 kB] 2020-07-23T12:19:17.189 INFO:teuthology.orchestra.run.smithi133.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [82.8 kB] 2020-07-23T12:19:17.195 INFO:teuthology.orchestra.run.smithi133.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [18.3 kB] 2020-07-23T12:19:17.196 INFO:teuthology.orchestra.run.smithi133.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,026 kB] 2020-07-23T12:19:17.241 INFO:teuthology.orchestra.run.smithi133.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,094 kB] 2020-07-23T12:19:17.282 INFO:teuthology.orchestra.run.smithi133.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [341 kB] 2020-07-23T12:19:17.873 INFO:teuthology.orchestra.run.smithi133.stdout:Fetched 8,076 kB in 2s (4,895 kB/s) 2020-07-23T12:19:19.218 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2020-07-23T12:19:19.259 INFO:teuthology.orchestra.run.smithi133:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-07-23T12:19:19.348 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2020-07-23T12:19:19.600 INFO:teuthology.orchestra.run.smithi133.stdout:Building dependency tree... 2020-07-23T12:19:19.600 INFO:teuthology.orchestra.run.smithi133.stdout:Reading state information... 2020-07-23T12:19:19.676 INFO:teuthology.orchestra.run.smithi133.stderr:E: Unable to locate package linux-image-current-generic 2020-07-23T12:19:19.678 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-07-23T12:19:19.679 INFO:teuthology.orchestra.run.smithi133:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-07-23T12:19:19.753 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2020-07-23T12:19:19.989 INFO:teuthology.orchestra.run.smithi133.stdout:Building dependency tree... 2020-07-23T12:19:19.990 INFO:teuthology.orchestra.run.smithi133.stdout:Reading state information... 2020-07-23T12:19:20.158 INFO:teuthology.orchestra.run.smithi133.stdout:The following additional packages will be installed: 2020-07-23T12:19:20.158 INFO:teuthology.orchestra.run.smithi133.stdout: linux-generic linux-headers-4.15.0-112 linux-headers-4.15.0-112-generic 2020-07-23T12:19:20.158 INFO:teuthology.orchestra.run.smithi133.stdout: linux-headers-generic linux-image-4.15.0-112-generic 2020-07-23T12:19:20.159 INFO:teuthology.orchestra.run.smithi133.stdout: linux-modules-4.15.0-112-generic linux-modules-extra-4.15.0-112-generic 2020-07-23T12:19:20.160 INFO:teuthology.orchestra.run.smithi133.stdout:Suggested packages: 2020-07-23T12:19:20.160 INFO:teuthology.orchestra.run.smithi133.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-07-23T12:19:20.160 INFO:teuthology.orchestra.run.smithi133.stdout:Recommended packages: 2020-07-23T12:19:20.161 INFO:teuthology.orchestra.run.smithi133.stdout: thermald 2020-07-23T12:19:20.201 INFO:teuthology.orchestra.run.smithi133.stdout:The following NEW packages will be installed: 2020-07-23T12:19:20.202 INFO:teuthology.orchestra.run.smithi133.stdout: linux-headers-4.15.0-112 linux-headers-4.15.0-112-generic 2020-07-23T12:19:20.202 INFO:teuthology.orchestra.run.smithi133.stdout: linux-image-4.15.0-112-generic linux-modules-4.15.0-112-generic 2020-07-23T12:19:20.203 INFO:teuthology.orchestra.run.smithi133.stdout: linux-modules-extra-4.15.0-112-generic 2020-07-23T12:19:20.203 INFO:teuthology.orchestra.run.smithi133.stdout:The following packages will be upgraded: 2020-07-23T12:19:20.203 INFO:teuthology.orchestra.run.smithi133.stdout: linux-generic linux-headers-generic linux-image-generic 2020-07-23T12:19:20.376 INFO:teuthology.orchestra.run.smithi133.stdout:3 upgraded, 5 newly installed, 0 to remove and 14 not upgraded. 2020-07-23T12:19:20.377 INFO:teuthology.orchestra.run.smithi133.stdout:Need to get 65.9 MB of archives. 2020-07-23T12:19:20.377 INFO:teuthology.orchestra.run.smithi133.stdout:After this operation, 333 MB of additional disk space will be used. 2020-07-23T12:19:20.378 INFO:teuthology.orchestra.run.smithi133.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-112-generic amd64 4.15.0-112.113 [13.1 MB] 2020-07-23T12:19:21.369 INFO:teuthology.orchestra.run.smithi133.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-112-generic amd64 4.15.0-112.113 [8,008 kB] 2020-07-23T12:19:21.557 INFO:teuthology.orchestra.run.smithi133.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-112-generic amd64 4.15.0-112.113 [32.8 MB] 2020-07-23T12:19:22.503 INFO:teuthology.orchestra.run.smithi133.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.112.100 [1,864 B] 2020-07-23T12:19:22.503 INFO:teuthology.orchestra.run.smithi133.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.112.100 [2,616 B] 2020-07-23T12:19:22.503 INFO:teuthology.orchestra.run.smithi133.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-112 all 4.15.0-112.113 [10.9 MB] 2020-07-23T12:19:22.815 INFO:teuthology.orchestra.run.smithi133.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-112-generic amd64 4.15.0-112.113 [1,089 kB] 2020-07-23T12:19:22.838 INFO:teuthology.orchestra.run.smithi133.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.112.100 [2,516 B] 2020-07-23T12:19:23.341 INFO:teuthology.orchestra.run.smithi133.stdout:Fetched 65.9 MB in 3s (25.1 MB/s) 2020-07-23T12:19:23.490 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package linux-modules-4.15.0-112-generic. 2020-07-23T12:19:23.526 INFO:teuthology.orchestra.run.smithi133.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102053 files and directories currently installed.) 2020-07-23T12:19:23.527 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../0-linux-modules-4.15.0-112-generic_4.15.0-112.113_amd64.deb ... 2020-07-23T12:19:23.578 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-modules-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:19:25.192 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package linux-image-4.15.0-112-generic. 2020-07-23T12:19:25.206 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../1-linux-image-4.15.0-112-generic_4.15.0-112.113_amd64.deb ... 2020-07-23T12:19:25.302 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-image-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:19:25.820 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-112-generic. 2020-07-23T12:19:25.832 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-112-generic_4.15.0-112.113_amd64.deb ... 2020-07-23T12:19:25.870 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-modules-extra-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:19:29.642 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../3-linux-generic_4.15.0.112.100_amd64.deb ... 2020-07-23T12:19:29.737 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-generic (4.15.0.112.100) over (4.15.0.111.99) ... 2020-07-23T12:19:30.071 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.112.100_amd64.deb ... 2020-07-23T12:19:30.194 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-image-generic (4.15.0.112.100) over (4.15.0.111.99) ... 2020-07-23T12:19:30.502 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package linux-headers-4.15.0-112. 2020-07-23T12:19:30.514 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../5-linux-headers-4.15.0-112_4.15.0-112.113_all.deb ... 2020-07-23T12:19:30.552 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-headers-4.15.0-112 (4.15.0-112.113) ... 2020-07-23T12:19:35.094 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package linux-headers-4.15.0-112-generic. 2020-07-23T12:19:35.123 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../6-linux-headers-4.15.0-112-generic_4.15.0-112.113_amd64.deb ... 2020-07-23T12:19:35.152 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-headers-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:19:37.538 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.112.100_amd64.deb ... 2020-07-23T12:19:37.635 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking linux-headers-generic (4.15.0.112.100) over (4.15.0.111.99) ... 2020-07-23T12:19:37.945 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-modules-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:19:38.482 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-headers-4.15.0-112 (4.15.0-112.113) ... 2020-07-23T12:19:38.611 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-image-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:19:39.068 INFO:teuthology.orchestra.run.smithi133.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-112-generic 2020-07-23T12:19:39.068 INFO:teuthology.orchestra.run.smithi133.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-112-generic 2020-07-23T12:19:39.185 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-modules-extra-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:19:39.670 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-headers-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:19:39.834 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-image-generic (4.15.0.112.100) ... 2020-07-23T12:19:39.955 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-headers-generic (4.15.0.112.100) ... 2020-07-23T12:19:40.081 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up linux-generic (4.15.0.112.100) ... 2020-07-23T12:19:40.207 INFO:teuthology.orchestra.run.smithi133.stdout:Processing triggers for linux-image-4.15.0-112-generic (4.15.0-112.113) ... 2020-07-23T12:19:40.424 INFO:teuthology.orchestra.run.smithi133.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-07-23T12:19:40.425 INFO:teuthology.orchestra.run.smithi133.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-112-generic 2020-07-23T12:19:55.697 INFO:teuthology.orchestra.run.smithi133.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-07-23T12:19:55.698 INFO:teuthology.orchestra.run.smithi133.stdout:Sourcing file `/etc/default/grub' 2020-07-23T12:19:55.724 INFO:teuthology.orchestra.run.smithi133.stdout:Generating grub configuration file ... 2020-07-23T12:19:56.072 INFO:teuthology.orchestra.run.smithi133.stdout:Found linux image: /boot/vmlinuz-4.15.0-112-generic 2020-07-23T12:19:56.114 INFO:teuthology.orchestra.run.smithi133.stdout:Found initrd image: /boot/initrd.img-4.15.0-112-generic 2020-07-23T12:19:56.443 INFO:teuthology.orchestra.run.smithi133.stdout:Found linux image: /boot/vmlinuz-4.15.0-111-generic 2020-07-23T12:19:56.470 INFO:teuthology.orchestra.run.smithi133.stdout:Found initrd image: /boot/initrd.img-4.15.0-111-generic 2020-07-23T12:19:57.948 INFO:teuthology.orchestra.run.smithi133.stdout:done 2020-07-23T12:19:58.371 INFO:teuthology.orchestra.run.smithi133:> dpkg -s linux-image-generic 2020-07-23T12:19:58.404 INFO:teuthology.orchestra.run.smithi133.stdout:Package: linux-image-generic 2020-07-23T12:19:58.404 INFO:teuthology.orchestra.run.smithi133.stdout:Status: install ok installed 2020-07-23T12:19:58.405 INFO:teuthology.orchestra.run.smithi133.stdout:Priority: optional 2020-07-23T12:19:58.405 INFO:teuthology.orchestra.run.smithi133.stdout:Section: kernel 2020-07-23T12:19:58.406 INFO:teuthology.orchestra.run.smithi133.stdout:Installed-Size: 16 2020-07-23T12:19:58.406 INFO:teuthology.orchestra.run.smithi133.stdout:Maintainer: Ubuntu Kernel Team 2020-07-23T12:19:58.407 INFO:teuthology.orchestra.run.smithi133.stdout:Architecture: amd64 2020-07-23T12:19:58.407 INFO:teuthology.orchestra.run.smithi133.stdout:Source: linux-meta 2020-07-23T12:19:58.408 INFO:teuthology.orchestra.run.smithi133.stdout:Version: 4.15.0.112.100 2020-07-23T12:19:58.408 INFO:teuthology.orchestra.run.smithi133.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.9) 2020-07-23T12:19:58.408 INFO:teuthology.orchestra.run.smithi133.stdout:Depends: linux-image-4.15.0-112-generic, linux-modules-extra-4.15.0-112-generic, linux-firmware, intel-microcode, amd64-microcode 2020-07-23T12:19:58.409 INFO:teuthology.orchestra.run.smithi133.stdout:Recommends: thermald 2020-07-23T12:19:58.409 INFO:teuthology.orchestra.run.smithi133.stdout:Description: Generic Linux kernel image 2020-07-23T12:19:58.409 INFO:teuthology.orchestra.run.smithi133.stdout: This package will always depend on the latest generic kernel image 2020-07-23T12:19:58.410 INFO:teuthology.orchestra.run.smithi133.stdout: available. 2020-07-23T12:19:58.411 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-111-generic Expected: 4.15.0-112-generic 2020-07-23T12:19:58.411 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-07-23T12:19:58.413 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-07-23T12:19:58.413 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-07-23T12:19:58.415 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2020-07-23T12:19:58.416 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2020-07-23T12:19:58.416 INFO:teuthology.orchestra.run.smithi137:> sudo apt-get clean 2020-07-23T12:19:58.486 INFO:teuthology.orchestra.run.smithi137:> sudo apt-get update 2020-07-23T12:19:58.734 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-07-23T12:19:58.741 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-07-23T12:19:58.802 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-07-23T12:19:58.881 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-07-23T12:20:00.589 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2020-07-23T12:20:00.634 INFO:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-07-23T12:20:00.713 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2020-07-23T12:20:00.943 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2020-07-23T12:20:00.944 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2020-07-23T12:20:01.020 INFO:teuthology.orchestra.run.smithi137.stderr:E: Unable to locate package linux-image-current-generic 2020-07-23T12:20:01.024 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-07-23T12:20:01.025 INFO:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-07-23T12:20:01.118 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2020-07-23T12:20:01.341 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2020-07-23T12:20:01.342 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2020-07-23T12:20:01.532 INFO:teuthology.orchestra.run.smithi137.stdout:linux-image-generic is already the newest version (4.15.0.112.100). 2020-07-23T12:20:01.533 INFO:teuthology.orchestra.run.smithi137.stdout:0 upgraded, 0 newly installed, 0 to remove and 14 not upgraded. 2020-07-23T12:20:01.533 INFO:teuthology.orchestra.run.smithi137:> dpkg -s linux-image-generic 2020-07-23T12:20:01.565 INFO:teuthology.orchestra.run.smithi137.stdout:Package: linux-image-generic 2020-07-23T12:20:01.566 INFO:teuthology.orchestra.run.smithi137.stdout:Status: install ok installed 2020-07-23T12:20:01.566 INFO:teuthology.orchestra.run.smithi137.stdout:Priority: optional 2020-07-23T12:20:01.567 INFO:teuthology.orchestra.run.smithi137.stdout:Section: kernel 2020-07-23T12:20:01.567 INFO:teuthology.orchestra.run.smithi137.stdout:Installed-Size: 16 2020-07-23T12:20:01.567 INFO:teuthology.orchestra.run.smithi137.stdout:Maintainer: Ubuntu Kernel Team 2020-07-23T12:20:01.567 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: amd64 2020-07-23T12:20:01.568 INFO:teuthology.orchestra.run.smithi137.stdout:Source: linux-meta 2020-07-23T12:20:01.568 INFO:teuthology.orchestra.run.smithi137.stdout:Version: 4.15.0.112.100 2020-07-23T12:20:01.568 INFO:teuthology.orchestra.run.smithi137.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.9) 2020-07-23T12:20:01.569 INFO:teuthology.orchestra.run.smithi137.stdout:Depends: linux-image-4.15.0-112-generic, linux-modules-extra-4.15.0-112-generic, linux-firmware, intel-microcode, amd64-microcode 2020-07-23T12:20:01.569 INFO:teuthology.orchestra.run.smithi137.stdout:Recommends: thermald 2020-07-23T12:20:01.570 INFO:teuthology.orchestra.run.smithi137.stdout:Description: Generic Linux kernel image 2020-07-23T12:20:01.570 INFO:teuthology.orchestra.run.smithi137.stdout: This package will always depend on the latest generic kernel image 2020-07-23T12:20:01.570 INFO:teuthology.orchestra.run.smithi137.stdout: available. 2020-07-23T12:20:01.571 INFO:teuthology.orchestra.run.smithi137:> mktemp 2020-07-23T12:20:01.642 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.xLk9svyi2X 2020-07-23T12:20:01.642 INFO:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub/grub.cfg /tmp/tmp.xLk9svyi2X 2020-07-23T12:20:01.726 INFO:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.xLk9svyi2X 2020-07-23T12:20:01.858 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.xLk9svyi2X is 8KB 2020-07-23T12:20:01.880 INFO:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.xLk9svyi2X 2020-07-23T12:20:01.895 INFO:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-07-23T12:20:01.946 INFO:teuthology.orchestra.run.smithi137:> sudo sh -c 'cat > /etc/grub.d/01_ceph_kernel' && sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-07-23T12:20:02.048 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-112-generic 2020-07-23T12:20:02.049 INFO:teuthology.orchestra.run.smithi137:> sudo update-grub 2020-07-23T12:20:03.358 INFO:teuthology.orchestra.run.smithi137.stderr:Sourcing file `/etc/default/grub' 2020-07-23T12:20:03.371 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2020-07-23T12:20:03.587 INFO:teuthology.orchestra.run.smithi137.stderr:Found linux image: /boot/vmlinuz-4.15.0-112-generic 2020-07-23T12:20:03.599 INFO:teuthology.orchestra.run.smithi137.stderr:Found initrd image: /boot/initrd.img-4.15.0-112-generic 2020-07-23T12:20:03.883 INFO:teuthology.orchestra.run.smithi137.stderr:Found linux image: /boot/vmlinuz-4.15.0-111-generic 2020-07-23T12:20:03.891 INFO:teuthology.orchestra.run.smithi137.stderr:Found initrd image: /boot/initrd.img-4.15.0-111-generic 2020-07-23T12:20:04.997 INFO:teuthology.orchestra.run.smithi137.stderr:done 2020-07-23T12:20:04.998 INFO:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2020-07-23T12:20:05.001 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2020-07-23T12:20:05.002 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi133.front.sepia.ceph.com, path=None, version=distro) 2020-07-23T12:20:05.002 INFO:teuthology.orchestra.run.smithi133:> sudo apt-get clean 2020-07-23T12:20:05.069 INFO:teuthology.orchestra.run.smithi133:> sudo apt-get update 2020-07-23T12:20:05.318 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-07-23T12:20:05.319 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-07-23T12:20:05.387 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-07-23T12:20:05.466 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-07-23T12:20:07.143 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2020-07-23T12:20:07.189 INFO:teuthology.orchestra.run.smithi133:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-07-23T12:20:07.265 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2020-07-23T12:20:07.499 INFO:teuthology.orchestra.run.smithi133.stdout:Building dependency tree... 2020-07-23T12:20:07.500 INFO:teuthology.orchestra.run.smithi133.stdout:Reading state information... 2020-07-23T12:20:07.578 INFO:teuthology.orchestra.run.smithi133.stderr:E: Unable to locate package linux-image-current-generic 2020-07-23T12:20:07.580 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-07-23T12:20:07.580 INFO:teuthology.orchestra.run.smithi133:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-07-23T12:20:07.660 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2020-07-23T12:20:07.910 INFO:teuthology.orchestra.run.smithi133.stdout:Building dependency tree... 2020-07-23T12:20:07.911 INFO:teuthology.orchestra.run.smithi133.stdout:Reading state information... 2020-07-23T12:20:08.101 INFO:teuthology.orchestra.run.smithi133.stdout:linux-image-generic is already the newest version (4.15.0.112.100). 2020-07-23T12:20:08.101 INFO:teuthology.orchestra.run.smithi133.stdout:0 upgraded, 0 newly installed, 0 to remove and 14 not upgraded. 2020-07-23T12:20:08.103 INFO:teuthology.orchestra.run.smithi133:> dpkg -s linux-image-generic 2020-07-23T12:20:08.127 INFO:teuthology.orchestra.run.smithi133.stdout:Package: linux-image-generic 2020-07-23T12:20:08.127 INFO:teuthology.orchestra.run.smithi133.stdout:Status: install ok installed 2020-07-23T12:20:08.127 INFO:teuthology.orchestra.run.smithi133.stdout:Priority: optional 2020-07-23T12:20:08.127 INFO:teuthology.orchestra.run.smithi133.stdout:Section: kernel 2020-07-23T12:20:08.128 INFO:teuthology.orchestra.run.smithi133.stdout:Installed-Size: 16 2020-07-23T12:20:08.128 INFO:teuthology.orchestra.run.smithi133.stdout:Maintainer: Ubuntu Kernel Team 2020-07-23T12:20:08.128 INFO:teuthology.orchestra.run.smithi133.stdout:Architecture: amd64 2020-07-23T12:20:08.128 INFO:teuthology.orchestra.run.smithi133.stdout:Source: linux-meta 2020-07-23T12:20:08.128 INFO:teuthology.orchestra.run.smithi133.stdout:Version: 4.15.0.112.100 2020-07-23T12:20:08.129 INFO:teuthology.orchestra.run.smithi133.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.9) 2020-07-23T12:20:08.129 INFO:teuthology.orchestra.run.smithi133.stdout:Depends: linux-image-4.15.0-112-generic, linux-modules-extra-4.15.0-112-generic, linux-firmware, intel-microcode, amd64-microcode 2020-07-23T12:20:08.129 INFO:teuthology.orchestra.run.smithi133.stdout:Recommends: thermald 2020-07-23T12:20:08.129 INFO:teuthology.orchestra.run.smithi133.stdout:Description: Generic Linux kernel image 2020-07-23T12:20:08.129 INFO:teuthology.orchestra.run.smithi133.stdout: This package will always depend on the latest generic kernel image 2020-07-23T12:20:08.130 INFO:teuthology.orchestra.run.smithi133.stdout: available. 2020-07-23T12:20:08.130 INFO:teuthology.orchestra.run.smithi133:> mktemp 2020-07-23T12:20:08.176 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.4dx1SHnXnA 2020-07-23T12:20:08.176 INFO:teuthology.orchestra.run.smithi133:> sudo cp /boot/grub/grub.cfg /tmp/tmp.4dx1SHnXnA 2020-07-23T12:20:08.233 INFO:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.4dx1SHnXnA 2020-07-23T12:20:08.350 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.4dx1SHnXnA is 8KB 2020-07-23T12:20:08.402 INFO:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.4dx1SHnXnA 2020-07-23T12:20:08.408 INFO:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-07-23T12:20:08.469 INFO:teuthology.orchestra.run.smithi133:> sudo sh -c 'cat > /etc/grub.d/01_ceph_kernel' && sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-07-23T12:20:08.575 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-112-generic 2020-07-23T12:20:08.575 INFO:teuthology.orchestra.run.smithi133:> sudo update-grub 2020-07-23T12:20:10.039 INFO:teuthology.orchestra.run.smithi133.stderr:Sourcing file `/etc/default/grub' 2020-07-23T12:20:10.062 INFO:teuthology.orchestra.run.smithi133.stderr:Generating grub configuration file ... 2020-07-23T12:20:10.278 INFO:teuthology.orchestra.run.smithi133.stderr:Found linux image: /boot/vmlinuz-4.15.0-112-generic 2020-07-23T12:20:10.290 INFO:teuthology.orchestra.run.smithi133.stderr:Found initrd image: /boot/initrd.img-4.15.0-112-generic 2020-07-23T12:20:10.562 INFO:teuthology.orchestra.run.smithi133.stderr:Found linux image: /boot/vmlinuz-4.15.0-111-generic 2020-07-23T12:20:10.578 INFO:teuthology.orchestra.run.smithi133.stderr:Found initrd image: /boot/initrd.img-4.15.0-111-generic 2020-07-23T12:20:11.666 INFO:teuthology.orchestra.run.smithi133.stderr:done 2020-07-23T12:20:11.669 INFO:teuthology.orchestra.run.smithi133:> sudo shutdown -r now 2020-07-23T12:20:41.703 INFO:teuthology.misc:Re-opening connections... 2020-07-23T12:20:41.704 INFO:teuthology.misc:trying to connect to ubuntu@smithi137.front.sepia.ceph.com 2020-07-23T12:20:41.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-23T12:20:41.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:21:41.713 DEBUG:teuthology.orchestra.remote:timed out 2020-07-23T12:21:41.717 INFO:teuthology.misc:trying to connect to ubuntu@smithi133.front.sepia.ceph.com 2020-07-23T12:21:41.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-23T12:21:41.720 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:21:44.798 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2020-07-23T12:21:44.799 DEBUG:teuthology.misc:waited 63.094440937042236 2020-07-23T12:21:45.799 INFO:teuthology.misc:trying to connect to ubuntu@smithi137.front.sepia.ceph.com 2020-07-23T12:21:45.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-23T12:21:45.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:21:46.206 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2020-07-23T12:21:46.207 INFO:teuthology.misc:trying to connect to ubuntu@smithi133.front.sepia.ceph.com 2020-07-23T12:21:46.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-23T12:21:46.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:21:47.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2020-07-23T12:21:47.871 DEBUG:teuthology.misc:waited 66.16614699363708 2020-07-23T12:21:48.871 INFO:teuthology.misc:trying to connect to ubuntu@smithi137.front.sepia.ceph.com 2020-07-23T12:21:48.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-23T12:21:48.873 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:21:48.968 INFO:teuthology.orchestra.run.smithi137:> true 2020-07-23T12:21:49.304 DEBUG:teuthology.misc:waited 67.59931325912476 2020-07-23T12:21:50.310 INFO:teuthology.misc:trying to connect to ubuntu@smithi133.front.sepia.ceph.com 2020-07-23T12:21:50.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-23T12:21:50.311 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:21:50.521 INFO:teuthology.orchestra.run.smithi133:> true 2020-07-23T12:21:51.619 DEBUG:teuthology.misc:waited 69.91451048851013 2020-07-23T12:21:52.621 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2020-07-23T12:21:52.621 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.15.0-112-generic"... 2020-07-23T12:21:52.622 INFO:teuthology.orchestra.run.smithi137:> uname -r 2020-07-23T12:21:52.628 INFO:teuthology.orchestra.run.smithi137.stdout:4.15.0-112-generic 2020-07-23T12:21:52.629 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-112-generic vs 4.15.0-112-generic 2020-07-23T12:21:52.629 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-07-23T12:21:52.629 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2020-07-23T12:21:52.630 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2020-07-23T12:21:52.630 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.15.0-112-generic"... 2020-07-23T12:21:52.631 INFO:teuthology.orchestra.run.smithi133:> uname -r 2020-07-23T12:21:52.638 INFO:teuthology.orchestra.run.smithi133.stdout:4.15.0-112-generic 2020-07-23T12:21:52.639 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-112-generic vs 4.15.0-112-generic 2020-07-23T12:21:52.640 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-07-23T12:21:52.640 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2020-07-23T12:21:53.641 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2020-07-23T12:21:53.642 INFO:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-07-23T12:21:53.699 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2020-07-23T12:21:53.717 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2020-07-23T12:21:53.718 INFO:teuthology.orchestra.run.smithi133:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-07-23T12:21:53.769 INFO:teuthology.orchestra.run.smithi133.stdout:ttyS1 2020-07-23T12:21:53.777 INFO:teuthology.run_tasks:Running task internal.base... 2020-07-23T12:21:53.796 INFO:teuthology.task.internal:Creating test directory... 2020-07-23T12:21:53.796 INFO:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-07-23T12:21:53.820 INFO:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-07-23T12:21:53.832 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-07-23T12:21:53.858 INFO:teuthology.run_tasks:Running task internal.archive... 2020-07-23T12:21:53.868 INFO:teuthology.task.internal:Creating archive directory... 2020-07-23T12:21:53.869 INFO:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-07-23T12:21:53.875 INFO:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-07-23T12:21:53.890 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-07-23T12:21:53.899 INFO:teuthology.task.internal:Enabling coredump saving... 2020-07-23T12:21:53.900 INFO:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-07-23T12:21:53.932 INFO:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-07-23T12:21:53.953 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-07-23T12:21:53.955 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-07-23T12:21:53.965 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-07-23T12:21:53.965 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-07-23T12:21:53.966 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-07-23T12:21:54.011 INFO:teuthology.task.internal:Configuring sudo... 2020-07-23T12:21:54.011 INFO:teuthology.orchestra.run.smithi133:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-07-23T12:21:54.017 INFO:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-07-23T12:21:54.057 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-07-23T12:21:54.080 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-07-23T12:21:54.082 INFO:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-07-23T12:21:54.085 INFO:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-07-23T12:21:54.105 INFO:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-07-23T12:21:54.157 INFO:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-07-23T12:21:54.208 INFO:teuthology.orchestra.run.smithi137:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-07-23T12:21:54.297 INFO:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-07-23T12:21:54.305 INFO:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-07-23T12:21:54.355 INFO:teuthology.orchestra.run.smithi133:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-07-23T12:21:54.467 INFO:teuthology.orchestra.run.smithi133:> sudo service rsyslog restart 2020-07-23T12:21:54.514 INFO:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2020-07-23T12:21:54.581 INFO:teuthology.run_tasks:Running task internal.timer... 2020-07-23T12:21:54.596 INFO:teuthology.task.internal:Starting timer... 2020-07-23T12:21:54.596 INFO:teuthology.run_tasks:Running task pcp... 2020-07-23T12:21:54.643 INFO:teuthology.run_tasks:Running task selinux... 2020-07-23T12:21:54.658 DEBUG:teuthology.task.selinux:Excluding smithi137: OS 'ubuntu' does not support SELinux 2020-07-23T12:21:54.662 DEBUG:teuthology.task.selinux:Excluding smithi133: OS 'ubuntu' does not support SELinux 2020-07-23T12:21:54.662 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-07-23T12:21:54.663 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-07-23T12:21:54.663 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-07-23T12:21:54.664 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-07-23T12:21:54.680 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-07-23T12:21:54.696 INFO:teuthology.repo_utils:Fetching master from origin 2020-07-23T12:21:54.788 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-07-23T12:21:54.811 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-07-23T12:21:54.815 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi137.front.sepia.ceph.com,smithi133.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-07-23T12:21:55.302 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-07-23T12:21:56.621 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-07-23T12:22:00.246 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-07-23T12:22:00.282 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:00 +0000 (0:00:00.112) 0:00:00.112 ********* 2020-07-23T12:22:00.306 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:00 +0000 (0:00:00.023) 0:00:00.136 ********* 2020-07-23T12:22:02.184 INFO:teuthology.task.ansible.out: 2020-07-23T12:22:02.186 INFO:teuthology.task.ansible.out:TASK [Gathering Facts] ********************************************************* 2020-07-23T12:22:02.188 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] 2020-07-23T12:22:02.265 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] 2020-07-23T12:22:02.309 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:02 +0000 (0:00:02.003) 0:00:02.139 ********* 2020-07-23T12:22:02.383 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:02 +0000 (0:00:00.074) 0:00:02.213 ********* 2020-07-23T12:22:02.738 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-07-23T12:22:02.792 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:02 +0000 (0:00:00.408) 0:00:02.622 ********* 2020-07-23T12:22:02.805 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-07-23T12:22:02.871 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:02 +0000 (0:00:00.078) 0:00:02.700 ********* 2020-07-23T12:22:03.581 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** changed: [smithi137.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-07-23T12:22:03.656 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:03 +0000 (0:00:00.782) 0:00:03.482 ********* 2020-07-23T12:22:03.680 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-07-23T12:22:03.733 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:03 +0000 (0:00:00.080) 0:00:03.563 ********* 2020-07-23T12:22:04.177 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.069643", "end": "2020-07-23 12:22:04.318196", "rc": 0, "start": "2020-07-23 12:22:04.248553", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-07-23T12:22:04.237 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:04 +0000 (0:00:00.503) 0:00:04.067 ********* 2020-07-23T12:22:04.250 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.078545", "end": "2020-07-23 12:22:04.260977", "rc": 0, "start": "2020-07-23 12:22:04.182432", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-07-23T12:22:04.334 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:04 +0000 (0:00:00.085) 0:00:04.153 ********* 2020-07-23T12:22:04.710 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi137.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-23T12:22:04.854 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:04 +0000 (0:00:00.531) 0:00:04.684 ********* 2020-07-23T12:22:04.864 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-23T12:22:04.917 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:04 +0000 (0:00:00.062) 0:00:04.747 ********* 2020-07-23T12:22:05.737 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1595506924.9365084-182763565892075/source", "state": "file", "uid": 0} 2020-07-23T12:22:05.805 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:05 +0000 (0:00:00.888) 0:00:05.635 ********* 2020-07-23T12:22:05.825 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1595506924.874304-151648519295766/source", "state": "file", "uid": 0} 2020-07-23T12:22:05.866 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:05 +0000 (0:00:00.060) 0:00:05.696 ********* 2020-07-23T12:22:06.235 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-07-23T12:22:06.237 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-07-23T12:22:06.238 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-07-23T12:22:06.273 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-07-23T12:22:06.275 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-07-23T12:22:06.399 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-07-23T12:22:06.414 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-07-23T12:22:06.555 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:06 +0000 (0:00:00.689) 0:00:06.385 ********* 2020-07-23T12:22:06.591 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:06 +0000 (0:00:00.036) 0:00:06.421 ********* 2020-07-23T12:22:07.302 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi133.front.sepia.ceph.com] 2020-07-23T12:22:07.313 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] 2020-07-23T12:22:07.422 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-07-23T12:22:07.424 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:07 +0000 (0:00:00.831) 0:00:07.253 ********* 2020-07-23T12:22:07.470 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:07 +0000 (0:00:00.046) 0:00:07.300 ********* 2020-07-23T12:22:07.862 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi133.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.198885", "end": "2020-07-23 12:22:07.830837", "rc": 0, "start": "2020-07-23 12:22:07.631952", "stderr": "", "stderr_lines": [], "stdout": "b65056c9880c09fd0e83d962e8d9709c27a571ac", "stdout_lines": ["b65056c9880c09fd0e83d962e8d9709c27a571ac"]} 2020-07-23T12:22:07.917 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:07 +0000 (0:00:00.446) 0:00:07.747 ********* 2020-07-23T12:22:07.919 INFO:teuthology.task.ansible.out: 2020-07-23T12:22:07.932 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.167737", "end": "2020-07-23 12:22:07.858273", "rc": 0, "start": "2020-07-23 12:22:07.690536", "stderr": "", "stderr_lines": [], "stdout": "b65056c9880c09fd0e83d962e8d9709c27a571ac", "stdout_lines": ["b65056c9880c09fd0e83d962e8d9709c27a571ac"]} 2020-07-23T12:22:07.934 INFO:teuthology.task.ansible.out: 2020-07-23T12:22:07.990 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:07 +0000 (0:00:00.073) 0:00:07.820 ********* 2020-07-23T12:22:08.114 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.022297", "end": "2020-07-23 12:22:08.246386", "failed_when_result": false, "rc": 0, "start": "2020-07-23 12:22:08.224089", "stderr": "", "stderr_lines": [], "stdout": "d2c541f972f084049b707e8806b2858a9ab4fb83", "stdout_lines": ["d2c541f972f084049b707e8806b2858a9ab4fb83"]} 2020-07-23T12:22:08.173 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:08 +0000 (0:00:00.182) 0:00:08.003 ********* 2020-07-23T12:22:08.210 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-07-23T12:22:08.212 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:08.249 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:08 +0000 (0:00:00.075) 0:00:08.079 ********* 2020-07-23T12:22:08.290 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:08.329 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:08 +0000 (0:00:00.080) 0:00:08.159 ********* 2020-07-23T12:22:08.370 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-07-23T12:22:08.372 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.216553", "end": "2020-07-23 12:22:08.436263", "failed_when_result": false, "rc": 0, "start": "2020-07-23 12:22:08.219710", "stderr": "", "stderr_lines": [], "stdout": "d2c541f972f084049b707e8806b2858a9ab4fb83", "stdout_lines": ["d2c541f972f084049b707e8806b2858a9ab4fb83"]} TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:08.408 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:08 +0000 (0:00:00.078) 0:00:08.238 ********* 2020-07-23T12:22:08.474 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:08 +0000 (0:00:00.060) 0:00:08.299 ********* 2020-07-23T12:22:08.478 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-07-23T12:22:08.480 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:08.526 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:08 +0000 (0:00:00.057) 0:00:08.356 ********* 2020-07-23T12:22:08.536 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-07-23T12:22:08.537 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:08.578 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:08 +0000 (0:00:00.050) 0:00:08.407 ********* 2020-07-23T12:22:08.587 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-07-23T12:22:08.589 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:08.627 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:08 +0000 (0:00:00.049) 0:00:08.457 ********* 2020-07-23T12:22:08.636 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-07-23T12:22:08.638 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:08.679 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:08 +0000 (0:00:00.052) 0:00:08.509 ********* 2020-07-23T12:22:08.729 INFO:teuthology.task.ansible.out: 2020-07-23T12:22:08.747 INFO:teuthology.task.ansible.out:TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-07-23T12:22:08.750 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:08.786 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:08 +0000 (0:00:00.107) 0:00:08.616 ********* 2020-07-23T12:22:08.796 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-07-23T12:22:08.802 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-23T12:22:08.838 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:08 +0000 (0:00:00.051) 0:00:08.668 ********* 2020-07-23T12:22:08.848 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-07-23T12:22:08.850 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:08.905 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:08 +0000 (0:00:00.066) 0:00:08.734 ********* 2020-07-23T12:22:08.949 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-07-23T12:22:08.952 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:08.997 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:08 +0000 (0:00:00.091) 0:00:08.827 ********* 2020-07-23T12:22:09.033 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-07-23T12:22:09.035 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-07-23T12:22:09.171 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-07-23T12:22:09.173 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-23T12:22:09.246 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:09 +0000 (0:00:00.247) 0:00:09.074 ********* 2020-07-23T12:22:09.257 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-07-23T12:22:09.259 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-07-23T12:22:09.357 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-07-23T12:22:09.470 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-07-23T12:22:09.522 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-07-23T12:22:09.622 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-07-23T12:22:09.662 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-07-23T12:22:09.759 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-07-23T12:22:09.807 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-07-23T12:22:09.893 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-07-23T12:22:09.992 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-07-23T12:22:10.024 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-07-23T12:22:10.133 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-07-23T12:22:10.187 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-07-23T12:22:10.298 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-07-23T12:22:10.339 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-07-23T12:22:10.445 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-07-23T12:22:10.467 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-07-23T12:22:10.619 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-07-23T12:22:10.665 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-07-23T12:22:10.754 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-07-23T12:22:10.817 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-07-23T12:22:10.921 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-07-23T12:22:10.964 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-07-23T12:22:11.085 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-07-23T12:22:11.132 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-07-23T12:22:11.238 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-07-23T12:22:11.286 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-07-23T12:22:11.404 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-07-23T12:22:11.437 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-07-23T12:22:11.552 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-07-23T12:22:11.607 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-07-23T12:22:11.687 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-07-23T12:22:11.797 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-07-23T12:22:11.834 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-07-23T12:22:11.960 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-07-23T12:22:12.003 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-07-23T12:22:12.131 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-07-23T12:22:12.158 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-07-23T12:22:12.315 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-07-23T12:22:12.329 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-07-23T12:22:12.474 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-07-23T12:22:12.475 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-07-23T12:22:12.633 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-07-23T12:22:12.635 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-07-23T12:22:12.799 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-07-23T12:22:12.958 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-07-23T12:22:12.974 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-07-23T12:22:13.132 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-07-23T12:22:13.144 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-07-23T12:22:13.312 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-07-23T12:22:13.325 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-07-23T12:22:13.469 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-07-23T12:22:13.536 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-07-23T12:22:13.633 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-07-23T12:22:13.710 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-07-23T12:22:13.779 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-07-23T12:22:13.851 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-07-23T12:22:13.938 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-07-23T12:22:14.025 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-07-23T12:22:14.090 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-07-23T12:22:14.173 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-07-23T12:22:14.238 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-07-23T12:22:14.339 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-07-23T12:22:14.400 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-07-23T12:22:14.514 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-07-23T12:22:14.553 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-07-23T12:22:14.674 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-07-23T12:22:14.713 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-07-23T12:22:14.829 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-07-23T12:22:14.872 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-07-23T12:22:15.001 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-07-23T12:22:15.024 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-07-23T12:22:15.161 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-07-23T12:22:15.173 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-07-23T12:22:15.295 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-07-23T12:22:15.342 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-07-23T12:22:15.458 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-07-23T12:22:15.491 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-07-23T12:22:15.610 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-07-23T12:22:15.665 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-07-23T12:22:15.754 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-07-23T12:22:15.843 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-07-23T12:22:15.926 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-07-23T12:22:15.998 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-07-23T12:22:16.093 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-07-23T12:22:16.180 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-07-23T12:22:16.262 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-07-23T12:22:16.332 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-07-23T12:22:16.408 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-07-23T12:22:16.484 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-07-23T12:22:16.575 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-07-23T12:22:16.630 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-07-23T12:22:16.779 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-07-23T12:22:16.794 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-07-23T12:22:16.940 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-07-23T12:22:16.964 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-07-23T12:22:17.147 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-07-23T12:22:17.150 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-07-23T12:22:17.333 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-07-23T12:22:17.485 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-07-23T12:22:17.510 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-07-23T12:22:17.646 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-07-23T12:22:17.658 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-07-23T12:22:17.813 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-07-23T12:22:17.819 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-07-23T12:22:17.955 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-07-23T12:22:17.986 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-07-23T12:22:18.125 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-07-23T12:22:18.170 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-07-23T12:22:18.300 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-07-23T12:22:18.333 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-07-23T12:22:18.335 INFO:teuthology.task.ansible.out: 2020-07-23T12:22:18.443 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-07-23T12:22:18.491 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-07-23T12:22:18.600 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-07-23T12:22:18.654 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-07-23T12:22:18.767 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-07-23T12:22:18.830 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-07-23T12:22:18.948 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-07-23T12:22:18.984 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-07-23T12:22:19.078 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-07-23T12:22:19.150 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-07-23T12:22:19.247 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-07-23T12:22:19.304 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-07-23T12:22:19.437 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-07-23T12:22:19.482 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-07-23T12:22:19.603 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-07-23T12:22:19.675 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-07-23T12:22:19.784 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-07-23T12:22:19.841 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-07-23T12:22:19.963 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-07-23T12:22:20.001 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-07-23T12:22:20.120 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-07-23T12:22:20.197 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-07-23T12:22:20.283 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-07-23T12:22:20.371 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-07-23T12:22:20.464 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-07-23T12:22:20.553 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-07-23T12:22:20.597 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-07-23T12:22:20.718 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-07-23T12:22:20.775 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-07-23T12:22:20.885 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-07-23T12:22:20.940 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-07-23T12:22:21.043 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-07-23T12:22:21.114 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-07-23T12:22:21.235 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-07-23T12:22:21.279 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-07-23T12:22:21.398 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-07-23T12:22:21.434 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-07-23T12:22:21.561 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-07-23T12:22:21.619 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-07-23T12:22:21.743 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-07-23T12:22:21.787 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-07-23T12:22:21.897 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-07-23T12:22:21.956 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-07-23T12:22:22.059 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-07-23T12:22:22.139 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-07-23T12:22:22.234 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-07-23T12:22:22.301 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-07-23T12:22:22.381 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-07-23T12:22:22.447 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-07-23T12:22:22.558 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-07-23T12:22:22.621 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-07-23T12:22:22.738 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-07-23T12:22:22.788 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-07-23T12:22:22.897 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-07-23T12:22:22.978 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-07-23T12:22:23.104 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-07-23T12:22:23.146 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-07-23T12:22:23.263 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-07-23T12:22:23.307 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-07-23T12:22:23.424 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-07-23T12:22:23.458 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-07-23T12:22:23.606 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-07-23T12:22:23.650 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-07-23T12:22:23.796 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-07-23T12:22:23.847 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-07-23T12:22:23.960 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-07-23T12:22:23.986 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-07-23T12:22:24.141 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-07-23T12:22:24.176 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-07-23T12:22:24.298 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-07-23T12:22:24.331 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-07-23T12:22:24.458 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-07-23T12:22:24.503 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-07-23T12:22:24.622 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-07-23T12:22:24.674 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-07-23T12:22:24.781 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-07-23T12:22:24.858 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-07-23T12:22:24.951 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-07-23T12:22:25.003 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-07-23T12:22:25.105 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-07-23T12:22:25.166 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-07-23T12:22:25.286 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-07-23T12:22:25.336 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-07-23T12:22:25.443 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-07-23T12:22:25.496 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-07-23T12:22:25.615 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-07-23T12:22:25.674 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-07-23T12:22:25.773 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-07-23T12:22:25.858 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-07-23T12:22:25.982 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-07-23T12:22:26.009 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-07-23T12:22:26.138 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-07-23T12:22:26.182 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-07-23T12:22:26.334 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-07-23T12:22:26.360 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-07-23T12:22:26.491 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-07-23T12:22:26.504 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-07-23T12:22:26.651 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-07-23T12:22:26.673 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-07-23T12:22:26.814 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-07-23T12:22:26.858 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-07-23T12:22:26.955 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-07-23T12:22:27.022 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-07-23T12:22:27.119 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-07-23T12:22:27.184 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-07-23T12:22:27.283 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2020-07-23T12:22:27.343 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-07-23T12:22:27.455 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2020-07-23T12:22:27.488 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2020-07-23T12:22:27.624 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2020-07-23T12:22:27.667 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2020-07-23T12:22:27.768 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2020-07-23T12:22:27.821 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2020-07-23T12:22:27.928 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2020-07-23T12:22:27.962 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2020-07-23T12:22:28.100 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2020-07-23T12:22:28.123 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2020-07-23T12:22:28.264 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2020-07-23T12:22:28.297 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2020-07-23T12:22:28.450 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2020-07-23T12:22:28.462 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2020-07-23T12:22:28.617 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2020-07-23T12:22:28.629 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2020-07-23T12:22:28.778 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2020-07-23T12:22:28.789 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2020-07-23T12:22:28.928 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2020-07-23T12:22:28.939 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2020-07-23T12:22:29.090 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2020-07-23T12:22:29.134 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2020-07-23T12:22:29.251 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2020-07-23T12:22:29.325 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2020-07-23T12:22:29.416 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2020-07-23T12:22:29.480 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2020-07-23T12:22:29.556 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2020-07-23T12:22:29.646 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2020-07-23T12:22:29.731 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2020-07-23T12:22:30.202 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-07-23T12:22:30.331 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-07-23T12:22:30.755 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2020-07-23T12:22:30.907 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:30 +0000 (0:00:21.662) 0:00:30.737 ********* 2020-07-23T12:22:30.920 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2020-07-23T12:22:31.001 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-07-23T12:22:31.038 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:31 +0000 (0:00:00.131) 0:00:30.868 ********* 2020-07-23T12:22:31.077 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:31 +0000 (0:00:00.038) 0:00:30.906 ********* 2020-07-23T12:22:31.112 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-07-23T12:22:31.114 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 2020-07-23T12:22:31.116 INFO:teuthology.task.ansible.out: 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2020-07-23T12:22:31.117 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivanci 2020-07-23T12:22:31.119 INFO:teuthology.task.ansible.out:ch", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0Wihhvk 2020-07-23T12:22:31.121 INFO:teuthology.task.ansible.out:KztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 2020-07-23T12:22:31.123 INFO:teuthology.task.ansible.out: HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfp 2020-07-23T12:22:31.124 INFO:teuthology.task.ansible.out:fKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1 2020-07-23T12:22:31.125 INFO:teuthology.task.ansible.out:GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73 2020-07-23T12:22:31.126 INFO:teuthology.task.ansible.out:dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localho 2020-07-23T12:22:31.128 INFO:teuthology.task.ansible.out:st eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn" 2020-07-23T12:22:31.130 INFO:teuthology.task.ansible.out:: "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"} 2020-07-23T12:22:31.131 INFO:teuthology.task.ansible.out:, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb 2020-07-23T12:22:31.133 INFO:teuthology.task.ansible.out:326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}]}, "changed": false} TASK [users : Create all users without sudo access.] *************************** 2020-07-23T12:22:31.177 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:31 +0000 (0:00:00.100) 0:00:31.007 ********* 2020-07-23T12:22:31.229 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:31 +0000 (0:00:00.052) 0:00:31.059 ********* 2020-07-23T12:22:31.296 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-07-23T12:22:31.303 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 2020-07-23T12:22:31.305 INFO:teuthology.task.ansible.out: 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2020-07-23T12:22:31.307 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivanci 2020-07-23T12:22:31.309 INFO:teuthology.task.ansible.out:ch", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e 2020-07-23T12:22:31.309 INFO:teuthology.task.ansible.out:39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0Wihhvk 2020-07-23T12:22:31.311 INFO:teuthology.task.ansible.out:KztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 2020-07-23T12:22:31.313 INFO:teuthology.task.ansible.out: HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfp 2020-07-23T12:22:31.315 INFO:teuthology.task.ansible.out:fKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73 2020-07-23T12:22:31.318 INFO:teuthology.task.ansible.out:dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localho 2020-07-23T12:22:31.320 INFO:teuthology.task.ansible.out:st eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn" 2020-07-23T12:22:31.321 INFO:teuthology.task.ansible.out:: "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"} 2020-07-23T12:22:31.323 INFO:teuthology.task.ansible.out:, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d 2020-07-23T12:22:31.324 INFO:teuthology.task.ansible.out:3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb 2020-07-23T12:22:31.325 INFO:teuthology.task.ansible.out:326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}]}, "changed": false} 2020-07-23T12:22:31.367 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:31 +0000 (0:00:00.137) 0:00:31.197 ********* 2020-07-23T12:22:31.369 INFO:teuthology.task.ansible.out: 2020-07-23T12:22:32.279 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* ok: [smithi137.front.sepia.ceph.com -> localhost] => {"after": "b65056c9880c09fd0e83d962e8d9709c27a571ac", "attempts": 1, "before": "b65056c9880c09fd0e83d962e8d9709c27a571ac", "changed": false, "remote_url_changed": false} ok: [smithi133.front.sepia.ceph.com -> localhost] => {"after": "b65056c9880c09fd0e83d962e8d9709c27a571ac", "attempts": 1, "before": "b65056c9880c09fd0e83d962e8d9709c27a571ac", "changed": false, "remote_url_changed": false} 2020-07-23T12:22:32.331 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:32 +0000 (0:00:00.964) 0:00:32.161 ********* 2020-07-23T12:22:32.371 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:32 +0000 (0:00:00.039) 0:00:32.201 ********* 2020-07-23T12:22:32.531 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-07-23T12:22:32.532 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-07-23T12:22:32.564 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-07-23T12:22:32.681 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-07-23T12:22:32.741 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-07-23T12:22:32.848 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-07-23T12:22:32.861 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-07-23T12:22:33.005 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2020-07-23T12:22:33.007 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2020-07-23T12:22:33.026 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2020-07-23T12:22:33.028 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2020-07-23T12:22:33.178 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-07-23T12:22:33.224 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-07-23T12:22:33.356 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2020-07-23T12:22:33.358 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-07-23T12:22:33.410 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2020-07-23T12:22:33.412 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-07-23T12:22:33.502 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-07-23T12:22:33.588 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-07-23T12:22:33.652 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2020-07-23T12:22:33.759 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2020-07-23T12:22:33.812 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-07-23T12:22:33.823 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:33.933 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-07-23T12:22:33.945 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:33.979 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2020-07-23T12:22:33.982 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-07-23T12:22:34.131 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2020-07-23T12:22:34.133 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-07-23T12:22:34.148 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2020-07-23T12:22:34.149 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2020-07-23T12:22:34.159 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:34.171 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:34.291 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2020-07-23T12:22:34.293 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2020-07-23T12:22:34.303 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:34.304 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:34.326 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-07-23T12:22:34.454 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-07-23T12:22:34.502 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2020-07-23T12:22:34.504 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-07-23T12:22:34.610 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2020-07-23T12:22:34.612 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-07-23T12:22:34.697 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-07-23T12:22:34.771 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-07-23T12:22:34.846 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-07-23T12:22:34.912 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-07-23T12:22:35.011 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2020-07-23T12:22:35.102 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2020-07-23T12:22:35.169 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-07-23T12:22:35.282 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-07-23T12:22:35.322 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-07-23T12:22:35.450 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-07-23T12:22:35.462 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-07-23T12:22:35.571 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-07-23T12:22:35.596 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2020-07-23T12:22:35.598 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588 2020-07-23T12:22:35.607 INFO:teuthology.task.ansible.out:Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-07-23T12:22:35.710 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2020-07-23T12:22:35.712 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-07-23T12:22:35.751 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2020-07-23T12:22:35.753 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-07-23T12:22:35.877 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2020-07-23T12:22:35.879 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-07-23T12:22:35.899 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-07-23T12:22:36.014 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-07-23T12:22:36.036 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-07-23T12:22:36.173 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-07-23T12:22:36.184 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2020-07-23T12:22:36.187 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-07-23T12:22:36.335 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-07-23T12:22:36.337 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2020-07-23T12:22:36.339 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-07-23T12:22:36.498 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-07-23T12:22:36.500 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-07-23T12:22:36.650 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-07-23T12:22:36.652 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-07-23T12:22:36.797 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-07-23T12:22:36.809 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-07-23T12:22:36.963 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-07-23T12:22:36.975 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-07-23T12:22:37.139 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2020-07-23T12:22:37.142 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-07-23T12:22:37.161 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-07-23T12:22:37.279 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-07-23T12:22:37.291 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2020-07-23T12:22:37.302 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-07-23T12:22:37.446 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-07-23T12:22:37.467 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-07-23T12:22:37.633 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-07-23T12:22:37.638 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-07-23T12:22:37.786 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-07-23T12:22:37.798 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-07-23T12:22:37.960 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-07-23T12:22:37.962 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-07-23T12:22:38.114 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-07-23T12:22:38.137 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-07-23T12:22:38.309 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-07-23T12:22:38.320 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-07-23T12:22:38.460 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-07-23T12:22:38.461 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-07-23T12:22:38.598 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-07-23T12:22:38.629 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-07-23T12:22:38.740 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2020-07-23T12:22:38.742 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-07-23T12:22:38.808 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-07-23T12:22:38.931 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2020-07-23T12:22:38.933 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-07-23T12:22:38.947 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2020-07-23T12:22:38.949 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-07-23T12:22:39.081 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-07-23T12:22:39.113 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2020-07-23T12:22:39.115 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-07-23T12:22:39.206 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-07-23T12:22:39.261 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-07-23T12:22:39.356 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-07-23T12:22:39.435 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-07-23T12:22:39.524 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2020-07-23T12:22:39.526 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-07-23T12:22:39.592 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-07-23T12:22:39.690 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2020-07-23T12:22:39.692 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2020-07-23T12:22:39.694 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-07-23T12:22:39.752 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2020-07-23T12:22:39.754 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-07-23T12:22:39.846 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-07-23T12:22:39.915 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2020-07-23T12:22:39.917 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2020-07-23T12:22:39.919 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-07-23T12:22:39.979 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-07-23T12:22:40.054 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-07-23T12:22:40.128 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-07-23T12:22:40.217 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-07-23T12:22:40.295 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-07-23T12:22:40.405 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-07-23T12:22:40.428 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-07-23T12:22:40.563 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-07-23T12:22:40.585 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2020-07-23T12:22:40.594 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-07-23T12:22:40.716 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-07-23T12:22:40.728 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-07-23T12:22:40.906 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-07-23T12:22:40.908 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2020-07-23T12:22:40.912 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-07-23T12:22:41.034 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2020-07-23T12:22:41.036 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2020-07-23T12:22:41.065 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-07-23T12:22:41.192 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-07-23T12:22:41.194 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-07-23T12:22:41.360 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-07-23T12:22:41.392 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2020-07-23T12:22:41.394 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2020-07-23T12:22:41.510 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2020-07-23T12:22:41.512 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-07-23T12:22:41.540 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-07-23T12:22:41.658 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-07-23T12:22:41.680 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-07-23T12:22:41.810 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-07-23T12:22:41.831 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2020-07-23T12:22:41.835 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-07-23T12:22:41.985 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-07-23T12:22:41.999 INFO:teuthology.task.ansible.out: ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-07-23T12:22:42.151 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-07-23T12:22:42.153 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-07-23T12:22:42.292 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-07-23T12:22:42.303 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-07-23T12:22:42.442 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-07-23T12:22:42.463 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-07-23T12:22:42.598 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-07-23T12:22:42.610 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-07-23T12:22:42.760 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-07-23T12:22:42.766 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-07-23T12:22:42.936 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-07-23T12:22:42.949 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-07-23T12:22:43.102 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-07-23T12:22:43.114 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-07-23T12:22:43.262 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-07-23T12:22:43.316 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-07-23T12:22:43.458 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-07-23T12:22:43.487 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-07-23T12:22:43.610 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-07-23T12:22:43.655 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-07-23T12:22:43.753 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-07-23T12:22:43.786 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-07-23T12:22:43.905 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-07-23T12:22:43.937 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-07-23T12:22:44.046 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-07-23T12:22:44.114 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-07-23T12:22:44.227 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-07-23T12:22:44.269 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-07-23T12:22:44.384 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-07-23T12:22:44.459 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-07-23T12:22:44.547 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-07-23T12:22:44.601 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-07-23T12:22:44.677 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-07-23T12:22:44.751 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-07-23T12:22:44.850 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-07-23T12:22:44.914 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-07-23T12:22:45.004 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-07-23T12:22:45.064 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-07-23T12:22:45.181 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-07-23T12:22:45.255 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-07-23T12:22:45.332 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-07-23T12:22:45.386 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-07-23T12:22:45.476 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-07-23T12:22:45.536 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-07-23T12:22:45.620 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2020-07-23T12:22:45.622 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-07-23T12:22:45.691 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-07-23T12:22:45.764 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2020-07-23T12:22:45.766 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-07-23T12:22:45.825 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-07-23T12:22:45.924 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-07-23T12:22:45.966 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2020-07-23T12:22:45.967 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-07-23T12:22:46.074 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-07-23T12:22:46.110 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2020-07-23T12:22:46.113 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-07-23T12:22:46.210 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-07-23T12:22:46.254 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-07-23T12:22:46.371 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-07-23T12:22:46.404 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-07-23T12:22:46.526 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-07-23T12:22:46.548 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-07-23T12:22:46.660 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-07-23T12:22:46.692 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-07-23T12:22:46.824 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-07-23T12:22:46.866 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-07-23T12:22:46.955 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-07-23T12:22:47.034 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-07-23T12:22:47.101 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-07-23T12:22:47.180 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-07-23T12:22:47.268 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-07-23T12:22:47.269 INFO:teuthology.task.ansible.out: 2020-07-23T12:22:47.321 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-07-23T12:22:47.406 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-07-23T12:22:47.480 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-07-23T12:22:47.591 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-07-23T12:22:47.634 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-07-23T12:22:47.745 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-07-23T12:22:47.795 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-07-23T12:22:47.904 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-07-23T12:22:47.937 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-07-23T12:22:48.082 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2020-07-23T12:22:48.085 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-07-23T12:22:48.116 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-07-23T12:22:48.265 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-07-23T12:22:48.306 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-07-23T12:22:48.424 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-07-23T12:22:48.482 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2020-07-23T12:22:48.484 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-07-23T12:22:48.589 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-07-23T12:22:48.657 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-07-23T12:22:48.762 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-07-23T12:22:48.807 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-07-23T12:22:48.923 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-07-23T12:22:48.966 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-07-23T12:22:49.088 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-07-23T12:22:49.131 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-07-23T12:22:49.267 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-07-23T12:22:49.278 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-07-23T12:22:49.444 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-07-23T12:22:49.456 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-07-23T12:22:49.603 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-07-23T12:22:49.619 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-07-23T12:22:49.777 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2020-07-23T12:22:49.780 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-07-23T12:22:49.783 INFO:teuthology.task.ansible.out: 2020-07-23T12:22:49.930 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-07-23T12:22:49.954 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2020-07-23T12:22:50.078 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2020-07-23T12:22:50.122 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2020-07-23T12:22:50.241 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2020-07-23T12:22:50.298 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2020-07-23T12:22:50.408 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2020-07-23T12:22:50.451 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2020-07-23T12:22:50.564 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2020-07-23T12:22:50.566 INFO:teuthology.task.ansible.out: 2020-07-23T12:22:50.610 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2020-07-23T12:22:50.712 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2020-07-23T12:22:50.780 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2020-07-23T12:22:50.869 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2020-07-23T12:22:50.934 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2020-07-23T12:22:50.999 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2020-07-23T12:22:51.084 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2020-07-23T12:22:51.164 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2020-07-23T12:22:51.250 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2020-07-23T12:22:51.253 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2020-07-23T12:22:51.315 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2020-07-23T12:22:51.399 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2020-07-23T12:22:51.486 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2020-07-23T12:22:51.487 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2020-07-23T12:22:51.558 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2020-07-23T12:22:51.620 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2020-07-23T12:22:51.716 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2020-07-23T12:22:51.791 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2020-07-23T12:22:51.866 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2020-07-23T12:22:51.963 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2020-07-23T12:22:52.028 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2020-07-23T12:22:52.177 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:52 +0000 (0:00:19.805) 0:00:52.007 ********* 2020-07-23T12:22:52.189 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2020-07-23T12:22:52.235 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-07-23T12:22:52.237 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.246 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.257 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.260 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.275 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.277 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi137.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2020-07-23T12:22:52.289 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-07-23T12:22:52.291 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.303 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-07-23T12:22:52.330 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.332 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.374 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:52 +0000 (0:00:00.197) 0:00:52.204 ********* 2020-07-23T12:22:52.428 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.440 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.442 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.458 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.463 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-07-23T12:22:52.478 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.479 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.480 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.492 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.498 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.503 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.514 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.643 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-07-23T12:22:52.665 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-07-23T12:22:52.699 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.701 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.810 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-07-23T12:22:52.812 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.822 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.833 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.852 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.861 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.864 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-07-23T12:22:52.874 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.876 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.885 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.919 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.921 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.921 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.935 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.937 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.947 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.958 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.960 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.970 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.987 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:52.994 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.002 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.004 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} ok: [smithi137.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+ 2020-07-23T12:22:53.006 INFO:teuthology.task.ansible.out:i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-07-23T12:22:53.039 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.042 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.044 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.059 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.073 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.079 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.085 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.096 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.098 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.110 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.132 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.139 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.141 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.141 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.142 INFO:teuthology.task.ansible.out: 2020-07-23T12:22:53.142 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.145 INFO:teuthology.task.ansible.out: skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.145 INFO:teuthology.task.ansible.out: 2020-07-23T12:22:53.157 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.159 INFO:teuthology.task.ansible.out: 2020-07-23T12:22:53.170 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.171 INFO:teuthology.task.ansible.out: skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.173 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.186 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.199 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.201 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.212 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.218 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.244 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.246 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.247 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.261 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.271 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.273 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbf 2020-07-23T12:22:53.274 INFO:teuthology.task.ansible.out:acffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.283 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.295 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.316 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.319 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.328 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.331 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.331 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwa 2020-07-23T12:22:53.333 INFO:teuthology.task.ansible.out:ng", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.342 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.344 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.355 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.360 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.368 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.370 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.379 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.381 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.391 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.393 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.402 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.404 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.414 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.416 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.426 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.428 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.438 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.440 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.450 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.452 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.462 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.464 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.474 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.485 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.487 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.502 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.511 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.512 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.513 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.514 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.525 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.527 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.538 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.539 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.541 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.553 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.555 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.566 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.568 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.579 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.581 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.592 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.595 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.605 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.607 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.608 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.629 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.631 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.642 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.644 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.654 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.665 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.666 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.677 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.678 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.689 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.691 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.702 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.713 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.715 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.716 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.717 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.728 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.730 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.751 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.754 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.754 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.755 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.766 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.767 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.769 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.781 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.783 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.784 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.795 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.796 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.797 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.819 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.821 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6 2020-07-23T12:22:53.823 INFO:teuthology.task.ansible.out:H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.833 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.835 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.846 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.861 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.907 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:53 +0000 (0:00:01.532) 0:00:53.736 ********* 2020-07-23T12:22:53.918 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.920 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.921 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.938 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.940 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.951 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.954 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:53.966 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-07-23T12:22:53.968 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.011 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:54 +0000 (0:00:00.104) 0:00:53.841 ********* 2020-07-23T12:22:54.021 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.024 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.024 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.045 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.047 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.058 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.061 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.070 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.081 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.083 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.092 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.094 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.104 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.119 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.131 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.133 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.143 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.144 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.156 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.167 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.178 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.190 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.279 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:54 +0000 (0:00:00.268) 0:00:54.109 ********* 2020-07-23T12:22:54.291 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-07-23T12:22:54.292 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-07-23T12:22:54.315 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-07-23T12:22:54.317 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:22:54.358 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:22:54 +0000 (0:00:00.079) 0:00:54.189 ********* 2020-07-23T12:22:54.370 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi133.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-07-23T12:22:54.492 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-07-23T12:22:54.563 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-07-23T12:22:54.663 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-07-23T12:22:54.696 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-07-23T12:22:54.796 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-07-23T12:22:54.870 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-07-23T12:22:54.959 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-07-23T12:22:55.009 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-07-23T12:22:55.080 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-07-23T12:22:55.152 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-07-23T12:22:55.211 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2020-07-23T12:22:55.276 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-07-23T12:22:55.345 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-07-23T12:22:55.424 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-07-23T12:22:55.499 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-07-23T12:22:55.575 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2020-07-23T12:22:55.653 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-07-23T12:22:55.707 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-07-23T12:22:55.807 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-07-23T12:22:55.829 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-07-23T12:22:55.958 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-07-23T12:22:55.960 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-07-23T12:22:56.098 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-07-23T12:22:56.109 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-07-23T12:22:56.240 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-07-23T12:22:56.272 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-07-23T12:22:56.401 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-07-23T12:22:56.426 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-07-23T12:22:56.532 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-07-23T12:22:56.599 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-07-23T12:22:56.669 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-07-23T12:22:56.744 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-07-23T12:22:56.825 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-07-23T12:22:56.893 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-07-23T12:22:56.949 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-07-23T12:22:57.041 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-07-23T12:22:57.102 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-07-23T12:22:57.202 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-07-23T12:22:57.289 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-07-23T12:22:57.372 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-07-23T12:22:57.422 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-07-23T12:22:57.534 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-07-23T12:22:57.570 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-07-23T12:22:57.682 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-07-23T12:22:57.729 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-07-23T12:22:57.836 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-07-23T12:22:57.858 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-07-23T12:22:58.021 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-07-23T12:22:58.034 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-07-23T12:22:58.149 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-07-23T12:22:58.187 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-07-23T12:22:58.282 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-07-23T12:22:58.330 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-07-23T12:22:58.463 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} ok: [smithi137.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-07-23T12:22:58.599 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-07-23T12:22:58.614 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-07-23T12:22:58.759 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-07-23T12:22:58.772 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-07-23T12:22:58.944 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-07-23T12:22:58.956 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-07-23T12:22:59.086 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} ok: [smithi137.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-07-23T12:22:59.250 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} ok: [smithi137.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-07-23T12:22:59.383 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-07-23T12:22:59.385 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-07-23T12:22:59.507 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-07-23T12:22:59.529 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-07-23T12:22:59.666 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-07-23T12:22:59.667 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-07-23T12:22:59.814 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-07-23T12:22:59.825 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-07-23T12:22:59.972 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-07-23T12:22:59.981 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-07-23T12:23:00.128 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-07-23T12:23:00.134 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-07-23T12:23:00.271 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-07-23T12:23:00.274 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-07-23T12:23:00.430 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-07-23T12:23:00.439 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-07-23T12:23:00.606 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-07-23T12:23:00.662 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-07-23T12:23:00.689 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:00 +0000 (0:00:06.330) 0:01:00.519 ********* 2020-07-23T12:23:00.768 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-07-23T12:23:00.828 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:00 +0000 (0:00:00.139) 0:01:00.658 ********* 2020-07-23T12:23:01.126 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "checksum": "2a4b5e34a9b23bcae760498dc647e0973285c643", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "bac924549c2094f640f3e6775820af27", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1595506980.725254-67217407555871/source", "state": "file", "uid": 0} 2020-07-23T12:23:01.182 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:01 +0000 (0:00:00.351) 0:01:01.010 ********* 2020-07-23T12:23:01.226 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi133.front.sepia.ceph.com] => { "msg": "Host smithi133.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-07-23T12:23:01.292 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:01 +0000 (0:00:00.103) 0:01:01.113 ********* 2020-07-23T12:23:01.297 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** changed: [smithi137.front.sepia.ceph.com] => {"changed": true, "checksum": "2a4b5e34a9b23bcae760498dc647e0973285c643", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "bac924549c2094f640f3e6775820af27", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1595506980.8745294-159227873240079/source", "state": "file", "uid": 0} 2020-07-23T12:23:01.357 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:01 +0000 (0:00:00.073) 0:01:01.187 ********* 2020-07-23T12:23:01.369 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-07-23T12:23:01.372 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-07-23T12:23:01.416 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:01 +0000 (0:00:00.059) 0:01:01.246 ********* 2020-07-23T12:23:01.429 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-07-23T12:23:01.432 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => { "msg": "Host smithi137.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-07-23T12:23:01.475 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:01 +0000 (0:00:00.058) 0:01:01.305 ********* 2020-07-23T12:23:01.491 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:01.551 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:01 +0000 (0:00:00.076) 0:01:01.381 ********* 2020-07-23T12:23:01.562 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-07-23T12:23:01.564 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-07-23T12:23:01.611 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:01 +0000 (0:00:00.060) 0:01:01.441 ********* 2020-07-23T12:23:01.623 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-07-23T12:23:01.626 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:01.668 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:01 +0000 (0:00:00.054) 0:01:01.496 ********* 2020-07-23T12:23:01.677 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:01.725 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:01 +0000 (0:00:00.058) 0:01:01.555 ********* 2020-07-23T12:23:01.735 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:01.781 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:01 +0000 (0:00:00.054) 0:01:01.610 ********* 2020-07-23T12:23:01.792 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:01.831 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:01 +0000 (0:00:00.051) 0:01:01.661 ********* 2020-07-23T12:23:01.841 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-07-23T12:23:01.843 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:01.883 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:01 +0000 (0:00:00.050) 0:01:01.712 ********* 2020-07-23T12:23:01.891 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-07-23T12:23:01.894 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:01.939 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:01 +0000 (0:00:00.056) 0:01:01.769 ********* 2020-07-23T12:23:01.948 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-07-23T12:23:01.950 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:01.990 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:01 +0000 (0:00:00.050) 0:01:01.820 ********* 2020-07-23T12:23:02.000 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-07-23T12:23:02.002 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:02.043 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:02 +0000 (0:00:00.053) 0:01:01.873 ********* 2020-07-23T12:23:02.053 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-07-23T12:23:02.056 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:02.120 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:02 +0000 (0:00:00.076) 0:01:01.950 ********* 2020-07-23T12:23:02.131 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:02.186 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:02 +0000 (0:00:00.066) 0:01:02.016 ********* 2020-07-23T12:23:02.196 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-07-23T12:23:02.198 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:02.236 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:02 +0000 (0:00:00.048) 0:01:02.064 ********* 2020-07-23T12:23:02.250 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:02.313 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:02 +0000 (0:00:00.071) 0:01:02.136 ********* 2020-07-23T12:23:02.319 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-07-23T12:23:02.322 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:02.372 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:02 +0000 (0:00:00.066) 0:01:02.202 ********* 2020-07-23T12:23:02.386 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:02.449 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:02 +0000 (0:00:00.076) 0:01:02.279 ********* 2020-07-23T12:23:02.463 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-07-23T12:23:02.465 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi133.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-23T12:23:02.531 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:02 +0000 (0:00:00.081) 0:01:02.361 ********* 2020-07-23T12:23:02.541 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-07-23T12:23:02.543 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:02.582 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:02 +0000 (0:00:00.048) 0:01:02.409 ********* 2020-07-23T12:23:02.591 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-07-23T12:23:02.592 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:02.644 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:02 +0000 (0:00:00.064) 0:01:02.474 ********* 2020-07-23T12:23:02.655 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-07-23T12:23:02.666 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi137.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-23T12:23:02.701 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:02 +0000 (0:00:00.054) 0:01:02.529 ********* 2020-07-23T12:23:02.712 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-07-23T12:23:02.725 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:02.763 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:02 +0000 (0:00:00.063) 0:01:02.592 ********* 2020-07-23T12:23:02.782 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:02.814 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:02 +0000 (0:00:00.051) 0:01:02.644 ********* 2020-07-23T12:23:02.824 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-07-23T12:23:02.826 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-23T12:23:02.871 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:02 +0000 (0:00:00.056) 0:01:02.700 ********* 2020-07-23T12:23:02.880 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-07-23T12:23:02.883 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:02.930 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:02 +0000 (0:00:00.056) 0:01:02.757 ********* 2020-07-23T12:23:02.942 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:02.990 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:02 +0000 (0:00:00.063) 0:01:02.820 ********* 2020-07-23T12:23:03.005 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-07-23T12:23:03.007 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-23T12:23:03.050 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.059) 0:01:02.880 ********* 2020-07-23T12:23:03.061 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi133.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-23T12:23:03.100 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.050) 0:01:02.930 ********* 2020-07-23T12:23:03.111 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-07-23T12:23:03.113 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:03.152 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.051) 0:01:02.982 ********* 2020-07-23T12:23:03.162 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-07-23T12:23:03.164 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:03.208 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.056) 0:01:03.038 ********* 2020-07-23T12:23:03.218 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-07-23T12:23:03.220 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-23T12:23:03.259 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.050) 0:01:03.089 ********* 2020-07-23T12:23:03.272 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:03.310 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.051) 0:01:03.140 ********* 2020-07-23T12:23:03.341 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:03.370 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.059) 0:01:03.199 ********* 2020-07-23T12:23:03.380 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-07-23T12:23:03.382 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:03.427 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.057) 0:01:03.257 ********* 2020-07-23T12:23:03.437 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-07-23T12:23:03.439 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:03.479 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.051) 0:01:03.309 ********* 2020-07-23T12:23:03.491 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-07-23T12:23:03.498 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:03.537 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.051) 0:01:03.360 ********* 2020-07-23T12:23:03.545 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-07-23T12:23:03.548 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:03.583 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.053) 0:01:03.414 ********* 2020-07-23T12:23:03.593 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-07-23T12:23:03.595 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-23T12:23:03.638 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.048) 0:01:03.462 ********* 2020-07-23T12:23:03.642 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-07-23T12:23:03.644 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:03.682 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.049) 0:01:03.511 ********* 2020-07-23T12:23:03.704 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:03.733 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.051) 0:01:03.563 ********* 2020-07-23T12:23:03.746 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi137.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-23T12:23:03.785 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.051) 0:01:03.614 ********* 2020-07-23T12:23:03.797 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-07-23T12:23:03.802 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:03.836 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.051) 0:01:03.666 ********* 2020-07-23T12:23:03.846 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-07-23T12:23:03.847 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:03.888 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.052) 0:01:03.718 ********* 2020-07-23T12:23:03.899 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-07-23T12:23:03.899 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:03.950 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:03 +0000 (0:00:00.061) 0:01:03.780 ********* 2020-07-23T12:23:03.962 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-07-23T12:23:03.963 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:04.007 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.057) 0:01:03.837 ********* 2020-07-23T12:23:04.016 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-07-23T12:23:04.018 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:04.069 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.062) 0:01:03.899 ********* 2020-07-23T12:23:04.082 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-07-23T12:23:04.085 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:04.126 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.056) 0:01:03.956 ********* 2020-07-23T12:23:04.136 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-07-23T12:23:04.138 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:04.179 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.052) 0:01:04.009 ********* 2020-07-23T12:23:04.190 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-07-23T12:23:04.192 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:04.229 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.048) 0:01:04.058 ********* 2020-07-23T12:23:04.238 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-07-23T12:23:04.238 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:04.295 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.066) 0:01:04.124 ********* 2020-07-23T12:23:04.308 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:04.357 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.062) 0:01:04.187 ********* 2020-07-23T12:23:04.366 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-07-23T12:23:04.369 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-23T12:23:04.406 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.047) 0:01:04.235 ********* 2020-07-23T12:23:04.416 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:04.457 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.051) 0:01:04.287 ********* 2020-07-23T12:23:04.467 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-07-23T12:23:04.469 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:04.511 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.054) 0:01:04.341 ********* 2020-07-23T12:23:04.524 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi137.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-23T12:23:04.581 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.069) 0:01:04.411 ********* 2020-07-23T12:23:04.592 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-07-23T12:23:04.602 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:04.636 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.055) 0:01:04.466 ********* 2020-07-23T12:23:04.647 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-07-23T12:23:04.649 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:04.713 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.072) 0:01:04.539 ********* 2020-07-23T12:23:04.723 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:04.778 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.065) 0:01:04.604 ********* 2020-07-23T12:23:04.787 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:04.832 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.057) 0:01:04.662 ********* 2020-07-23T12:23:04.842 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-07-23T12:23:04.844 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:04.896 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.062) 0:01:04.725 ********* 2020-07-23T12:23:04.918 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:04.946 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.051) 0:01:04.776 ********* 2020-07-23T12:23:04.955 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-07-23T12:23:04.958 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-07-23T12:23:05.002 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:04 +0000 (0:00:00.055) 0:01:04.832 ********* 2020-07-23T12:23:05.022 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:05.039 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:05.077 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:05 +0000 (0:00:00.074) 0:01:04.907 ********* 2020-07-23T12:23:05.088 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:05.132 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:05 +0000 (0:00:00.052) 0:01:04.960 ********* 2020-07-23T12:23:05.146 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* skipping: [smithi137.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-07-23T12:23:05.198 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:05 +0000 (0:00:00.067) 0:01:05.028 ********* 2020-07-23T12:23:05.242 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-07-23T12:23:05.244 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:05.283 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:05 +0000 (0:00:00.084) 0:01:05.112 ********* 2020-07-23T12:23:05.390 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-07-23T12:23:05.392 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.077811", "end": "2020-07-23 12:23:05.500096", "rc": 0, "start": "2020-07-23 12:23:05.422285", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-07-23T12:23:05.478 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:05 +0000 (0:00:00.195) 0:01:05.307 ********* 2020-07-23T12:23:05.505 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.052930", "end": "2020-07-23 12:23:05.561745", "rc": 0, "start": "2020-07-23 12:23:05.508815", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-07-23T12:23:05.508 INFO:teuthology.task.ansible.out: 2020-07-23T12:23:05.554 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:05 +0000 (0:00:00.067) 0:01:05.375 ********* 2020-07-23T12:23:05.559 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:05.612 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:05 +0000 (0:00:00.065) 0:01:05.441 ********* 2020-07-23T12:23:05.626 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:05.694 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:05 +0000 (0:00:00.070) 0:01:05.511 ********* TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:05.762 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:05 +0000 (0:00:00.073) 0:01:05.585 ********* 2020-07-23T12:23:05.771 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:05.823 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:05 +0000 (0:00:00.067) 0:01:05.653 ********* 2020-07-23T12:23:05.834 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:05.891 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:05 +0000 (0:00:00.068) 0:01:05.721 ********* 2020-07-23T12:23:05.908 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:05.945 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:05 +0000 (0:00:00.053) 0:01:05.775 ********* 2020-07-23T12:23:05.959 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:06.010 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:06 +0000 (0:00:00.064) 0:01:05.840 ********* 2020-07-23T12:23:06.021 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:06.061 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:06 +0000 (0:00:00.051) 0:01:05.891 ********* 2020-07-23T12:23:06.074 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:06.119 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:06 +0000 (0:00:00.057) 0:01:05.949 ********* 2020-07-23T12:23:06.130 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:06.180 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:06 +0000 (0:00:00.060) 0:01:06.009 ********* 2020-07-23T12:23:06.200 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:06.253 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:06 +0000 (0:00:00.074) 0:01:06.083 ********* 2020-07-23T12:23:06.267 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:06.309 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:06 +0000 (0:00:00.056) 0:01:06.140 ********* 2020-07-23T12:23:06.326 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:06.378 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:06 +0000 (0:00:00.068) 0:01:06.208 ********* 2020-07-23T12:23:06.396 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:06.438 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:06 +0000 (0:00:00.057) 0:01:06.266 ********* 2020-07-23T12:23:07.386 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.834307", "end": "2020-07-23 12:23:07.500466", "rc": 0, "start": "2020-07-23 12:23:06.666159", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:23:07.440 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:07 +0000 (0:00:01.004) 0:01:07.270 ********* 2020-07-23T12:23:07.452 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.840971", "end": "2020-07-23 12:23:07.500363", "rc": 0, "start": "2020-07-23 12:23:06.659392", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:23:07.492 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:07 +0000 (0:00:00.051) 0:01:07.322 ********* 2020-07-23T12:23:07.503 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-07-23T12:23:07.507 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:07.555 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:07 +0000 (0:00:00.063) 0:01:07.385 ********* 2020-07-23T12:23:07.567 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:07.610 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:07 +0000 (0:00:00.054) 0:01:07.439 ********* 2020-07-23T12:23:07.761 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-07-23T12:23:07.763 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.041880", "end": "2020-07-23 12:23:07.875553", "rc": 0, "start": "2020-07-23 12:23:07.833673", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:23:07.814 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:07 +0000 (0:00:00.203) 0:01:07.643 ********* 2020-07-23T12:23:07.840 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.051829", "end": "2020-07-23 12:23:07.882153", "rc": 0, "start": "2020-07-23 12:23:07.830324", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:23:07.879 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:07 +0000 (0:00:00.065) 0:01:07.709 ********* 2020-07-23T12:23:11.498 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** ok: [smithi133.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1595506758, "cache_updated": false, "changed": false} 2020-07-23T12:23:11.539 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:11 +0000 (0:00:03.659) 0:01:11.369 ********* 2020-07-23T12:23:11.681 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1595506715, "cache_updated": false, "changed": false} 2020-07-23T12:23:11.727 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:11 +0000 (0:00:00.188) 0:01:11.557 ********* 2020-07-23T12:23:12.383 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-07-23T12:23:12.385 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1595506758, "cache_updated": false, "changed": false} 2020-07-23T12:23:12.437 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:12 +0000 (0:00:00.709) 0:01:12.267 ********* 2020-07-23T12:23:12.483 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:12.545 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:12 +0000 (0:00:00.108) 0:01:12.375 ********* 2020-07-23T12:23:12.592 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-07-23T12:23:12.594 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"cache_update_time": 1595506715, "cache_updated": false, "changed": false} 2020-07-23T12:23:12.648 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:12 +0000 (0:00:00.102) 0:01:12.477 ********* 2020-07-23T12:23:12.702 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:12.753 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:12 +0000 (0:00:00.105) 0:01:12.583 ********* 2020-07-23T12:23:13.314 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-07-23T12:23:13.318 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-07-23T12:23:13.358 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:13 +0000 (0:00:00.604) 0:01:13.188 ********* 2020-07-23T12:23:13.403 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:13 +0000 (0:00:00.045) 0:01:13.233 ********* 2020-07-23T12:23:14.247 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-07-23T12:23:14.444 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"cache_update_time": 1595506715, "cache_updated": false, "changed": false} Thursday 23 July 2020 12:23:14 +0000 (0:00:00.896) 0:01:14.130 ********* ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1595506758, "cache_updated": false, "changed": false} Thursday 23 July 2020 12:23:14 +0000 (0:00:00.053) 0:01:14.183 ********* 2020-07-23T12:23:14.799 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi137.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-07-23T12:23:14.805 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-07-23T12:23:15.144 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-07-23T12:23:15.202 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-07-23T12:23:15.508 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-07-23T12:23:15.573 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:15 +0000 (0:00:01.219) 0:01:15.403 ********* 2020-07-23T12:23:15.585 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-07-23T12:23:15.633 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:15 +0000 (0:00:00.060) 0:01:15.463 ********* 2020-07-23T12:23:15.730 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-07-23T12:23:15.732 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-07-23T12:23:15.776 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:15 +0000 (0:00:00.142) 0:01:15.606 ********* 2020-07-23T12:23:15.798 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-07-23T12:23:15.842 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:15 +0000 (0:00:00.065) 0:01:15.672 ********* 2020-07-23T12:23:16.193 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-07-23T12:23:16.195 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-07-23T12:23:16.227 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-07-23T12:23:16.547 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-07-23T12:23:16.592 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-07-23T12:23:16.890 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-07-23T12:23:16.944 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:16 +0000 (0:00:01.101) 0:01:16.774 ********* 2020-07-23T12:23:16.956 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-07-23T12:23:16.999 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:16 +0000 (0:00:00.055) 0:01:16.829 ********* 2020-07-23T12:23:17.014 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-07-23T12:23:17.016 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-07-23T12:23:17.066 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:17 +0000 (0:00:00.066) 0:01:16.896 ********* 2020-07-23T12:23:17.078 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-07-23T12:23:17.130 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:17 +0000 (0:00:00.057) 0:01:16.954 ********* 2020-07-23T12:23:17.134 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:17.177 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:17 +0000 (0:00:00.050) 0:01:17.004 ********* 2020-07-23T12:23:17.186 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:17.226 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:17 +0000 (0:00:00.052) 0:01:17.057 ********* 2020-07-23T12:23:17.246 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* skipping: [smithi137.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-23T12:23:17.283 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:17 +0000 (0:00:00.056) 0:01:17.113 ********* 2020-07-23T12:23:17.295 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-23T12:23:17.356 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:17 +0000 (0:00:00.071) 0:01:17.185 ********* 2020-07-23T12:23:17.370 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi137.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-23T12:23:17.381 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-23T12:23:17.392 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-23T12:23:17.448 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:17 +0000 (0:00:00.093) 0:01:17.279 ********* 2020-07-23T12:23:17.459 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-23T12:23:17.461 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-07-23T12:23:17.500 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:17 +0000 (0:00:00.051) 0:01:17.330 ********* 2020-07-23T12:23:17.510 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-07-23T12:23:17.512 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:17.556 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:17 +0000 (0:00:00.056) 0:01:17.386 ********* 2020-07-23T12:23:17.568 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:17.618 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:17 +0000 (0:00:00.061) 0:01:17.448 ********* 2020-07-23T12:23:18.524 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-07-23T12:23:18.527 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"cache_update_time": 1595506715, "cache_updated": false, "changed": false} 2020-07-23T12:23:18.570 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:18 +0000 (0:00:00.947) 0:01:18.396 ********* 2020-07-23T12:23:18.590 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1595506758, "cache_updated": false, "changed": false} 2020-07-23T12:23:18.628 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:18 +0000 (0:00:00.062) 0:01:18.458 ********* 2020-07-23T12:23:18.963 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-07-23T12:23:19.056 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:19 +0000 (0:00:00.427) 0:01:18.886 ********* 2020-07-23T12:23:19.067 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-07-23T12:23:19.112 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:19 +0000 (0:00:00.056) 0:01:18.942 ********* 2020-07-23T12:23:19.514 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-07-23T12:23:19.519 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-07-23T12:23:19.571 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:19 +0000 (0:00:00.459) 0:01:19.401 ********* 2020-07-23T12:23:19.582 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-07-23T12:23:19.635 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:19 +0000 (0:00:00.063) 0:01:19.465 ********* 2020-07-23T12:23:19.736 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ ok: [smithi133.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-07-23T12:23:19.813 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-07-23T12:23:19.859 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-07-23T12:23:19.906 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:19 +0000 (0:00:00.270) 0:01:19.736 ********* 2020-07-23T12:23:19.942 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-07-23T12:23:19.946 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:19.989 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:19 +0000 (0:00:00.083) 0:01:19.819 ********* 2020-07-23T12:23:20.027 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:20 +0000 (0:00:00.038) 0:01:19.857 ********* 2020-07-23T12:23:20.036 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-07-23T12:23:20.038 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-23T12:23:20.078 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:20 +0000 (0:00:00.050) 0:01:19.908 ********* 2020-07-23T12:23:20.089 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-07-23T12:23:20.091 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:20.141 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:20 +0000 (0:00:00.063) 0:01:19.971 ********* 2020-07-23T12:23:20.152 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-07-23T12:23:20.153 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:20.192 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:20 +0000 (0:00:00.051) 0:01:20.022 ********* 2020-07-23T12:23:20.202 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-07-23T12:23:20.204 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-07-23T12:23:20.242 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:20 +0000 (0:00:00.049) 0:01:20.072 ********* 2020-07-23T12:23:20.284 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-07-23T12:23:20.285 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:20.331 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:20 +0000 (0:00:00.089) 0:01:20.162 ********* 2020-07-23T12:23:20.600 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-07-23T12:23:20.602 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-07-23T12:23:20.655 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:20 +0000 (0:00:00.323) 0:01:20.485 ********* 2020-07-23T12:23:20.725 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-07-23T12:23:20.773 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:20 +0000 (0:00:00.118) 0:01:20.603 ********* 2020-07-23T12:23:21.450 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-07-23T12:23:21.451 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-07-23 12:21:50 UTC", "ActiveEnterTimestampMonotonic": "20593727", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target system.slice nss-lookup.target -.mount time-sync.target systemd-tmpfiles-setup.service systemd-journald.socket basic.target var-run.mount sysinit.target local-fs.target remote-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-07-23 12:21:50 UTC", "AssertTimestampMonotonic": "20592792", "Before": "xdm.service multi-user.target shutdown.target plymouth-quit.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-07-23 12:21:50 UTC", "ConditionTimestampMonotonic": "20592791", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-07-23T12:23:21.453 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1318", "ExecMainStartTimestamp": "Thu 2020-07-23 12:21:50 UTC", "ExecMainStartTimestampMonotonic": "20593695", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2020-07-23 12:21:50 UTC] ; stop_time=[n/a] ; pid=1318 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-07-23 12:21:50 UTC", "InactiveExitTimestampMonotonic": "20593727", "InvocationID": "27efa6d5920241f38f6a9975bfec5490", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2020-07-23T12:23:21.454 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1318", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2020-07-23T12:23:21.457 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-07-23 12:21:50 UTC", "StateChangeTimestampMonotonic": "20593727", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-07-23 12:21:50 UTC", "WatchdogTimestampMonotonic": "20593727", "WatchdogUSec": "0"}} 2020-07-23T12:23:21.534 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:21 +0000 (0:00:00.760) 0:01:21.363 ********* 2020-07-23T12:23:21.548 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-07-23 12:21:46 UTC", "ActiveEnterTimestampMonotonic": "21132773", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target basic.target system.slice time-sync.target -.mount var-run.mount remote-fs.target local-fs.target systemd-tmpfiles-setup.service nss-lookup.target network.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-07-23 12:21:46 UTC", "AssertTimestampMonotonic": "21130475", "Before": "shutdown.target plymouth-quit.service xdm.service multi-user.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-07-23 12:21:46 UTC", "ConditionTimestampMonotonic": "21130474", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-07-23T12:23:21.550 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1429", "ExecMainStartTimestamp": "Thu 2020-07-23 12:21:46 UTC", "ExecMainStartTimestampMonotonic": "21132717", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2020-07-23 12:21:46 UTC] ; stop_time=[n/a] ; pid=1429 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-07-23 12:21:46 UTC", "InactiveExitTimestampMonotonic": "21132773", "InvocationID": "bd7d37724bc7414fbdb921ee8287186a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2020-07-23T12:23:21.551 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1429", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2020-07-23T12:23:21.553 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-07-23 12:21:46 UTC", "StateChangeTimestampMonotonic": "21132773", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-07-23 12:21:46 UTC", "WatchdogTimestampMonotonic": "21132772", "WatchdogUSec": "0"}} 2020-07-23T12:23:21.591 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:21 +0000 (0:00:00.057) 0:01:21.421 ********* 2020-07-23T12:23:21.604 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-07-23T12:23:21.606 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:21.646 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:21 +0000 (0:00:00.055) 0:01:21.476 ********* 2020-07-23T12:23:21.660 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:21.716 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:21 +0000 (0:00:00.069) 0:01:21.546 ********* 2020-07-23T12:23:21.738 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:21.788 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:21 +0000 (0:00:00.071) 0:01:21.618 ********* 2020-07-23T12:23:21.801 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:21.840 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:21 +0000 (0:00:00.052) 0:01:21.670 ********* 2020-07-23T12:23:21.852 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-07-23T12:23:21.854 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:21.892 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:21 +0000 (0:00:00.051) 0:01:21.722 ********* 2020-07-23T12:23:21.902 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:21.941 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:21 +0000 (0:00:00.048) 0:01:21.771 ********* 2020-07-23T12:23:21.980 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-07-23T12:23:21.981 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:22.019 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:22 +0000 (0:00:00.077) 0:01:21.849 ********* 2020-07-23T12:23:22.029 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:22.068 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:22 +0000 (0:00:00.048) 0:01:21.898 ********* 2020-07-23T12:23:22.080 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-07-23T12:23:22.082 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:22.125 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:22 +0000 (0:00:00.057) 0:01:21.955 ********* 2020-07-23T12:23:22.141 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:22.183 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:22 +0000 (0:00:00.057) 0:01:22.013 ********* 2020-07-23T12:23:22.309 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-07-23T12:23:22.311 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:22.345 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:22 +0000 (0:00:00.162) 0:01:22.175 ********* 2020-07-23T12:23:22.355 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:22.395 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:22 +0000 (0:00:00.049) 0:01:22.225 ********* 2020-07-23T12:23:22.404 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-07-23T12:23:22.406 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:22.443 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:22 +0000 (0:00:00.048) 0:01:22.273 ********* 2020-07-23T12:23:22.453 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:22.491 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:22 +0000 (0:00:00.048) 0:01:22.322 ********* 2020-07-23T12:23:22.504 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:22.540 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:22 +0000 (0:00:00.048) 0:01:22.370 ********* 2020-07-23T12:23:22.550 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:22.590 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:22 +0000 (0:00:00.049) 0:01:22.420 ********* 2020-07-23T12:23:22.600 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-07-23T12:23:22.602 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:22.641 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:22 +0000 (0:00:00.050) 0:01:22.471 ********* 2020-07-23T12:23:22.651 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:22.710 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:22 +0000 (0:00:00.069) 0:01:22.541 ********* 2020-07-23T12:23:22.721 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-07-23T12:23:22.723 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:22.762 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:22 +0000 (0:00:00.051) 0:01:22.592 ********* 2020-07-23T12:23:22.772 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:22.810 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:22 +0000 (0:00:00.047) 0:01:22.640 ********* 2020-07-23T12:23:22.818 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-07-23T12:23:22.820 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-07-23T12:23:22.868 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:22 +0000 (0:00:00.058) 0:01:22.698 ********* 2020-07-23T12:23:22.878 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-07-23T12:23:22.917 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:22 +0000 (0:00:00.049) 0:01:22.747 ********* 2020-07-23T12:23:22.925 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-07-23T12:23:22.936 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-07-23T12:23:22.970 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:22 +0000 (0:00:00.051) 0:01:22.799 ********* 2020-07-23T12:23:22.982 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-07-23T12:23:23.022 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:23 +0000 (0:00:00.052) 0:01:22.851 ********* 2020-07-23T12:23:23.029 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-07-23T12:23:23.031 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-07-23T12:23:23.071 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:23 +0000 (0:00:00.049) 0:01:22.901 ********* 2020-07-23T12:23:23.082 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-07-23T12:23:23.130 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:23 +0000 (0:00:00.051) 0:01:22.952 ********* 2020-07-23T12:23:23.135 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi133.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-07-23T12:23:23.202 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:23 +0000 (0:00:00.078) 0:01:23.031 ********* 2020-07-23T12:23:23.212 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-07-23T12:23:23.272 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:23 +0000 (0:00:00.062) 0:01:23.094 ********* 2020-07-23T12:23:23.363 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-07-23T12:23:23.417 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:23 +0000 (0:00:00.153) 0:01:23.247 ********* 2020-07-23T12:23:23.429 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-07-23T12:23:23.494 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:23 +0000 (0:00:00.077) 0:01:23.324 ********* 2020-07-23T12:23:23.580 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-07-23T12:23:23.582 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 1224, "name": "ubuntu", "state": "present", "system": false} 2020-07-23T12:23:23.626 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:23 +0000 (0:00:00.131) 0:01:23.456 ********* 2020-07-23T12:23:23.663 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "gid": 1224, "name": "ubuntu", "state": "present", "system": false} 2020-07-23T12:23:23.696 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:23 +0000 (0:00:00.070) 0:01:23.526 ********* 2020-07-23T12:23:23.706 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-07-23T12:23:23.708 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:23.751 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:23 +0000 (0:00:00.049) 0:01:23.576 ********* 2020-07-23T12:23:23.759 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:23.802 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:23 +0000 (0:00:00.055) 0:01:23.632 ********* 2020-07-23T12:23:23.814 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-07-23T12:23:23.815 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:23.852 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:23 +0000 (0:00:00.049) 0:01:23.681 ********* 2020-07-23T12:23:23.863 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:23.906 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:23 +0000 (0:00:00.053) 0:01:23.735 ********* 2020-07-23T12:23:24.031 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-07-23T12:23:24.033 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-07-23T12:23:24.069 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:24 +0000 (0:00:00.163) 0:01:23.899 ********* 2020-07-23T12:23:24.100 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-07-23T12:23:24.152 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:24 +0000 (0:00:00.082) 0:01:23.982 ********* 2020-07-23T12:23:24.568 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-07-23T12:23:24.570 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-07-23T12:23:24.611 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:24 +0000 (0:00:00.458) 0:01:24.441 ********* 2020-07-23T12:23:24.652 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-07-23T12:23:24.664 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi133"}, "changed": false} 2020-07-23T12:23:24.704 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:24 +0000 (0:00:00.093) 0:01:24.535 ********* 2020-07-23T12:23:24.757 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:24 +0000 (0:00:00.052) 0:01:24.587 ********* 2020-07-23T12:23:24.801 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi137"}, "changed": false} 2020-07-23T12:23:24.843 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:24 +0000 (0:00:00.085) 0:01:24.673 ********* 2020-07-23T12:23:25.608 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ changed: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi133.front.sepia.ceph.com", "ansible_hostname": "smithi133", "ansible_nodename": "smithi133"}, "changed": true, "name": "smithi133"} 2020-07-23T12:23:25.673 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:25 +0000 (0:00:00.829) 0:01:25.503 ********* 2020-07-23T12:23:25.845 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi137.front.sepia.ceph.com", "ansible_hostname": "smithi137", "ansible_nodename": "smithi137"}, "changed": true, "name": "smithi137"} 2020-07-23T12:23:25.904 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:25 +0000 (0:00:00.231) 0:01:25.734 ********* 2020-07-23T12:23:26.077 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-07-23T12:23:26.135 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:26 +0000 (0:00:00.231) 0:01:25.965 ********* 2020-07-23T12:23:26.322 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-07-23T12:23:26.402 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:26 +0000 (0:00:00.266) 0:01:26.232 ********* 2020-07-23T12:23:26.533 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-07-23T12:23:26.566 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:26 +0000 (0:00:00.164) 0:01:26.397 ********* 2020-07-23T12:23:26.798 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-07-23T12:23:26.844 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:26 +0000 (0:00:00.277) 0:01:26.674 ********* 2020-07-23T12:23:27.087 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-07-23T12:23:27.089 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-07-23T12:23:27.161 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:27 +0000 (0:00:00.317) 0:01:26.991 ********* 2020-07-23T12:23:27.206 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-07-23T12:23:27.208 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:27.248 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:27 +0000 (0:00:00.087) 0:01:27.078 ********* 2020-07-23T12:23:27.280 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-07-23T12:23:27.282 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:27.320 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:27 +0000 (0:00:00.071) 0:01:27.150 ********* 2020-07-23T12:23:27.357 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Add ssh pubkeys] ********************************************** changed: [smithi137.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-07-23T12:23:27.400 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:27 +0000 (0:00:00.073) 0:01:27.224 ********* 2020-07-23T12:23:27.439 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:27 +0000 (0:00:00.044) 0:01:27.269 ********* 2020-07-23T12:23:27.451 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:27.500 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:27 +0000 (0:00:00.060) 0:01:27.330 ********* 2020-07-23T12:23:27.522 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:27.586 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:27 +0000 (0:00:00.082) 0:01:27.412 ********* 2020-07-23T12:23:27.597 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:27.647 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:27 +0000 (0:00:00.062) 0:01:27.474 ********* 2020-07-23T12:23:27.657 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-07-23T12:23:27.659 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:27.706 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:27 +0000 (0:00:00.061) 0:01:27.536 ********* 2020-07-23T12:23:27.720 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:27.759 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:27 +0000 (0:00:00.053) 0:01:27.589 ********* 2020-07-23T12:23:27.769 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-07-23T12:23:27.771 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:27.809 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:27 +0000 (0:00:00.050) 0:01:27.640 ********* 2020-07-23T12:23:27.819 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-07-23T12:23:27.820 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:27.887 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:27 +0000 (0:00:00.071) 0:01:27.711 ********* 2020-07-23T12:23:27.894 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-07-23T12:23:27.895 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:27.938 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:27 +0000 (0:00:00.056) 0:01:27.768 ********* 2020-07-23T12:23:27.951 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:27.991 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:27 +0000 (0:00:00.053) 0:01:27.821 ********* 2020-07-23T12:23:28.004 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-07-23T12:23:28.006 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:28.044 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:28 +0000 (0:00:00.052) 0:01:27.874 ********* 2020-07-23T12:23:28.056 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-07-23T12:23:28.106 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:28 +0000 (0:00:00.061) 0:01:27.936 ********* 2020-07-23T12:23:28.119 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-07-23T12:23:28.120 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:28.166 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:28 +0000 (0:00:00.057) 0:01:27.994 ********* 2020-07-23T12:23:28.174 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-07-23T12:23:28.227 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:28 +0000 (0:00:00.063) 0:01:28.057 ********* 2020-07-23T12:23:28.237 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-07-23T12:23:28.239 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:28.278 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:28 +0000 (0:00:00.050) 0:01:28.108 ********* 2020-07-23T12:23:28.292 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-07-23T12:23:28.343 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:28 +0000 (0:00:00.065) 0:01:28.173 ********* 2020-07-23T12:23:28.354 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-07-23T12:23:28.356 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:28.395 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:28 +0000 (0:00:00.052) 0:01:28.225 ********* 2020-07-23T12:23:28.408 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-07-23T12:23:28.410 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:28.454 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:28 +0000 (0:00:00.059) 0:01:28.284 ********* 2020-07-23T12:23:28.466 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-07-23T12:23:28.516 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:28 +0000 (0:00:00.061) 0:01:28.346 ********* 2020-07-23T12:23:28.529 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-07-23T12:23:28.531 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-07-23T12:23:28.572 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:28 +0000 (0:00:00.056) 0:01:28.402 ********* 2020-07-23T12:23:28.590 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-07-23T12:23:28.626 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:28 +0000 (0:00:00.053) 0:01:28.456 ********* 2020-07-23T12:23:28.638 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-07-23T12:23:28.640 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:28.679 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:28 +0000 (0:00:00.052) 0:01:28.509 ********* 2020-07-23T12:23:28.692 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-07-23T12:23:28.733 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:28 +0000 (0:00:00.053) 0:01:28.563 ********* 2020-07-23T12:23:28.746 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-23T12:23:28.748 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:28.793 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:28 +0000 (0:00:00.060) 0:01:28.623 ********* 2020-07-23T12:23:28.807 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-07-23T12:23:28.809 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:28.848 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:28 +0000 (0:00:00.054) 0:01:28.678 ********* 2020-07-23T12:23:28.858 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-07-23T12:23:28.860 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:28.905 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:28 +0000 (0:00:00.057) 0:01:28.735 ********* 2020-07-23T12:23:28.917 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-07-23T12:23:28.919 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-07-23T12:23:28.959 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:28 +0000 (0:00:00.054) 0:01:28.790 ********* 2020-07-23T12:23:28.969 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-07-23T12:23:28.971 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.012 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.052) 0:01:28.842 ********* 2020-07-23T12:23:29.022 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-07-23T12:23:29.024 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.061 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.049) 0:01:28.891 ********* 2020-07-23T12:23:29.070 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-07-23T12:23:29.072 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.115 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.053) 0:01:28.945 ********* 2020-07-23T12:23:29.125 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-23T12:23:29.127 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.164 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.048) 0:01:28.994 ********* 2020-07-23T12:23:29.174 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-07-23T12:23:29.176 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.212 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.047) 0:01:29.042 ********* 2020-07-23T12:23:29.222 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-07-23T12:23:29.224 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.264 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.052) 0:01:29.095 ********* 2020-07-23T12:23:29.276 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.320 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.055) 0:01:29.150 ********* 2020-07-23T12:23:29.332 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-07-23T12:23:29.334 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.372 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.052) 0:01:29.202 ********* 2020-07-23T12:23:29.383 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-07-23T12:23:29.384 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.426 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.051) 0:01:29.254 ********* 2020-07-23T12:23:29.437 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-07-23T12:23:29.438 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.477 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.052) 0:01:29.307 ********* 2020-07-23T12:23:29.487 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-07-23T12:23:29.489 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.543 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.066) 0:01:29.373 ********* 2020-07-23T12:23:29.557 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-07-23T12:23:29.559 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.607 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.064) 0:01:29.437 ********* 2020-07-23T12:23:29.620 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-07-23T12:23:29.622 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.664 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.056) 0:01:29.494 ********* 2020-07-23T12:23:29.676 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-07-23T12:23:29.678 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.725 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.061) 0:01:29.555 ********* 2020-07-23T12:23:29.736 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-07-23T12:23:29.737 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.780 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.054) 0:01:29.609 ********* 2020-07-23T12:23:29.790 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-07-23T12:23:29.792 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.829 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.049) 0:01:29.659 ********* 2020-07-23T12:23:29.840 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-07-23T12:23:29.842 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.883 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.053) 0:01:29.713 ********* 2020-07-23T12:23:29.892 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-07-23T12:23:29.894 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.933 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.050) 0:01:29.763 ********* 2020-07-23T12:23:29.945 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-07-23T12:23:29.946 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:29.983 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:29 +0000 (0:00:00.049) 0:01:29.813 ********* 2020-07-23T12:23:29.993 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-07-23T12:23:29.994 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:30.033 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.049) 0:01:29.863 ********* 2020-07-23T12:23:30.043 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:30.081 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.048) 0:01:29.911 ********* 2020-07-23T12:23:30.091 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-07-23T12:23:30.093 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:30.137 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.053) 0:01:29.965 ********* 2020-07-23T12:23:30.144 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-07-23T12:23:30.146 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:30.183 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.048) 0:01:30.013 ********* 2020-07-23T12:23:30.193 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-07-23T12:23:30.194 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:30.234 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.051) 0:01:30.064 ********* 2020-07-23T12:23:30.244 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-07-23T12:23:30.246 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:30.282 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.047) 0:01:30.112 ********* 2020-07-23T12:23:30.294 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:30.333 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.050) 0:01:30.163 ********* 2020-07-23T12:23:30.347 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-07-23T12:23:30.349 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:30.389 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.054) 0:01:30.218 ********* 2020-07-23T12:23:30.397 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-07-23T12:23:30.400 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:30.439 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.051) 0:01:30.269 ********* 2020-07-23T12:23:30.449 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-07-23T12:23:30.451 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:30.489 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.050) 0:01:30.319 ********* 2020-07-23T12:23:30.510 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:30.545 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.053) 0:01:30.373 ********* 2020-07-23T12:23:30.554 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-07-23T12:23:30.555 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:30.598 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.054) 0:01:30.428 ********* 2020-07-23T12:23:30.609 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-07-23T12:23:30.611 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:30.649 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.050) 0:01:30.479 ********* 2020-07-23T12:23:30.661 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:30.700 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.051) 0:01:30.530 ********* 2020-07-23T12:23:30.712 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-07-23T12:23:30.713 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:30.753 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.052) 0:01:30.583 ********* 2020-07-23T12:23:30.805 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-07-23T12:23:30.806 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:30.843 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.090) 0:01:30.673 ********* 2020-07-23T12:23:30.875 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-07-23T12:23:30.877 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.003338", "end": "2020-07-23 12:23:30.962826", "rc": 0, "start": "2020-07-23 12:23:30.959488", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:23:30.927 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.083) 0:01:30.757 ********* 2020-07-23T12:23:30.937 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-07-23T12:23:30.939 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:30.979 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:30 +0000 (0:00:00.051) 0:01:30.809 ********* 2020-07-23T12:23:30.992 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-07-23T12:23:31.027 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:31 +0000 (0:00:00.048) 0:01:30.857 ********* 2020-07-23T12:23:31.037 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-07-23T12:23:31.039 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:31.083 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:31 +0000 (0:00:00.055) 0:01:30.913 ********* 2020-07-23T12:23:31.231 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-07-23T12:23:31.233 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005199", "end": "2020-07-23 12:23:31.279688", "rc": 0, "start": "2020-07-23 12:23:31.274489", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:23:31.288 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:31 +0000 (0:00:00.205) 0:01:31.118 ********* 2020-07-23T12:23:31.319 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-07-23T12:23:31.357 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:31 +0000 (0:00:00.068) 0:01:31.187 ********* 2020-07-23T12:23:31.398 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-07-23T12:23:31.400 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-07-23T12:23:31.445 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:31 +0000 (0:00:00.088) 0:01:31.275 ********* 2020-07-23T12:23:31.486 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-07-23T12:23:31.488 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:31.524 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:31 +0000 (0:00:00.078) 0:01:31.354 ********* 2020-07-23T12:23:31.724 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-07-23T12:23:31.768 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:31 +0000 (0:00:00.243) 0:01:31.598 ********* 2020-07-23T12:23:31.805 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-07-23T12:23:31.806 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:31.850 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:31 +0000 (0:00:00.079) 0:01:31.677 ********* 2020-07-23T12:23:32.846 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-07-23T12:23:32.848 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-07-23T12:23:32.900 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-07-23T12:23:33.764 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-07-23T12:23:33.810 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:33 +0000 (0:00:01.962) 0:01:33.640 ********* 2020-07-23T12:23:33.843 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-07-23T12:23:33.890 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:33 +0000 (0:00:00.079) 0:01:33.720 ********* 2020-07-23T12:23:34.732 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* ok: [smithi137.front.sepia.ceph.com] => {"cache_update_time": 1595506715, "cache_updated": false, "changed": false} 2020-07-23T12:23:34.779 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:34 +0000 (0:00:00.889) 0:01:34.609 ********* 2020-07-23T12:23:34.813 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-07-23T12:23:34.862 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:34 +0000 (0:00:00.082) 0:01:34.691 ********* 2020-07-23T12:23:34.883 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-07-23T12:23:34.886 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1595506758, "cache_updated": false, "changed": false} 2020-07-23T12:23:34.933 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:34 +0000 (0:00:00.071) 0:01:34.763 ********* 2020-07-23T12:23:34.968 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-07-23T12:23:35.014 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:35 +0000 (0:00:00.080) 0:01:34.844 ********* 2020-07-23T12:23:35.068 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-07-23T12:23:35.070 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi137.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.036152", "end": "2020-07-23 12:23:35.104927", "rc": 0, "start": "2020-07-23 12:23:35.068775", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:23:35.126 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:35 +0000 (0:00:00.112) 0:01:34.956 ********* 2020-07-23T12:23:35.214 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.030750", "end": "2020-07-23 12:23:35.317690", "rc": 0, "start": "2020-07-23 12:23:35.286940", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:23:35.257 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:35 +0000 (0:00:00.131) 0:01:35.087 ********* 2020-07-23T12:23:38.063 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-07-23T12:23:38.065 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"cache_update_time": 1595506715, "cache_updated": false, "changed": false} 2020-07-23T12:23:38.137 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:38 +0000 (0:00:02.876) 0:01:37.964 ********* 2020-07-23T12:23:38.253 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1595506758, "cache_updated": false, "changed": false} 2020-07-23T12:23:38.305 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:38 +0000 (0:00:00.171) 0:01:38.135 ********* 2020-07-23T12:23:46.737 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-07-23T12:23:46.739 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 14 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137578 files and directorie 2020-07-23T12:23:46.741 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 14 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2020-07-23T12:23:46.743 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137578 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-07-23T12:23:46.791 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:46 +0000 (0:00:08.483) 0:01:46.618 ********* 2020-07-23T12:23:46.943 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 14 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137578 files and directorie 2020-07-23T12:23:46.945 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 14 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2020-07-23T12:23:46.947 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137578 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-07-23T12:23:47.006 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:46 +0000 (0:00:00.216) 0:01:46.835 ********* 2020-07-23T12:23:47.729 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-07-23T12:23:47.731 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"cache_update_time": 1595506715, "cache_updated": false, "changed": false} 2020-07-23T12:23:47.797 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:47 +0000 (0:00:00.792) 0:01:47.627 ********* 2020-07-23T12:23:47.830 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1595506758, "cache_updated": false, "changed": false} 2020-07-23T12:23:47.871 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:47 +0000 (0:00:00.073) 0:01:47.701 ********* 2020-07-23T12:23:47.883 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-07-23T12:23:47.885 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:47.927 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:47 +0000 (0:00:00.056) 0:01:47.757 ********* 2020-07-23T12:23:47.939 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:47.987 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:47 +0000 (0:00:00.059) 0:01:47.817 ********* 2020-07-23T12:23:56.505 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-07-23T12:23:56.508 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => {"cache_update_time": 1595506715, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 14 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.2 [2723 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.2 [923 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.28 [39.6 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.28 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.28 [5188 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.28 [870 kB]\nFetched 10.4 MB in 1s (9810 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2020-07-23T12:23:56.510 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137436 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.2_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.2_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2020-07-23T12:23:56.512 INFO:teuthology.task.ansible.out:u1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 14 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.2 [2723 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.2 [923 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.28 [39.6 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.28 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.28 [5188 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.28 [870 kB]", "Fetched 10.4 MB in 1s (9810 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2020-07-23T12:23:56.514 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137436 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.2_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.2_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-07-23T12:23:56.556 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:56 +0000 (0:00:08.569) 0:01:56.386 ********* 2020-07-23T12:23:56.613 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1595506758, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 14 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.2 [2723 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.2 [923 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.28 [39.6 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.28 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.28 [5188 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.28 [870 kB]\nFetched 10.4 MB in 1s (9850 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2020-07-23T12:23:56.615 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137436 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.2_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.2_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2020-07-23T12:23:56.617 INFO:teuthology.task.ansible.out:u1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 14 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.2 [2723 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.2 [923 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.28 [39.6 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.28 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.28 [5188 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.28 [870 kB]", "Fetched 10.4 MB in 1s (9850 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2020-07-23T12:23:56.619 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137436 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.2_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.2_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.28_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.28) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.28) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-07-23T12:23:56.656 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:56 +0000 (0:00:00.098) 0:01:56.485 ********* 2020-07-23T12:23:57.554 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-07-23T12:23:57.556 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1595506758, "cache_updated": false, "changed": false} 2020-07-23T12:23:57.606 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:57 +0000 (0:00:00.949) 0:01:57.435 ********* 2020-07-23T12:23:57.632 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"cache_update_time": 1595506715, "cache_updated": false, "changed": false} 2020-07-23T12:23:57.684 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:57 +0000 (0:00:00.079) 0:01:57.514 ********* 2020-07-23T12:23:57.764 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-07-23T12:23:57.765 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-07-23T12:23:57.828 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:57 +0000 (0:00:00.143) 0:01:57.658 ********* 2020-07-23T12:23:57.870 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-07-23T12:23:57.871 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-07-23T12:23:57.871 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:57.925 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:57 +0000 (0:00:00.096) 0:01:57.755 ********* 2020-07-23T12:23:57.996 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:57 +0000 (0:00:00.071) 0:01:57.826 ********* 2020-07-23T12:23:58.031 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:58.071 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:58 +0000 (0:00:00.074) 0:01:57.901 ********* 2020-07-23T12:23:58.093 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-07-23T12:23:58.134 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:58 +0000 (0:00:00.063) 0:01:57.965 ********* 2020-07-23T12:23:58.137 INFO:teuthology.task.ansible.out: 2020-07-23T12:23:58.232 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-07-23T12:23:58.273 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:58 +0000 (0:00:00.138) 0:01:58.103 ********* 2020-07-23T12:23:58.534 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-07-23T12:23:58.542 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-07-23T12:23:58.592 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:58 +0000 (0:00:00.318) 0:01:58.422 ********* 2020-07-23T12:23:58.699 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-07-23T12:23:58.748 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:58 +0000 (0:00:00.155) 0:01:58.577 ********* 2020-07-23T12:23:58.782 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ ok: [smithi133.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1224, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-07-23T12:23:58.839 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:58 +0000 (0:00:00.091) 0:01:58.669 ********* 2020-07-23T12:23:58.940 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1224, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-07-23T12:23:58.992 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:58 +0000 (0:00:00.152) 0:01:58.821 ********* 2020-07-23T12:23:59.020 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-07-23T12:23:59.022 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-23T12:23:59.069 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:59 +0000 (0:00:00.077) 0:01:58.899 ********* 2020-07-23T12:23:59.163 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-23T12:23:59.210 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:59 +0000 (0:00:00.137) 0:01:59.036 ********* 2020-07-23T12:23:59.417 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2020-07-23 12:21:50 UTC", "ActiveEnterTimestampMonotonic": "20948857", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target basic.target network.target systemd-tmpfiles-setup.service sysinit.target system.slice -.mount systemd-journald.socket remote-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-07-23 12:21:50 UTC", "AssertTimestampMonotonic": "20595655", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-07-23 12:21:50 UTC", "ConditionTimestampMonotonic": "20595655", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", 2020-07-23T12:23:59.437 INFO:teuthology.task.ansible.out:"Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", "DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1407", "ExecMainStartTimestamp": "Thu 2020-07-23 12:21:50 UTC", "ExecMainStartTimestampMonotonic": "20948846", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-07-23 12:21:50 UTC", "InactiveExitTimestampMonotonic": "20596960", "InvocationID": "658cdd5d6a41415a94d2edd173feadeb", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA 2020-07-23T12:23:59.439 INFO:teuthology.task.ansible.out:": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1407", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", 2020-07-23T12:23:59.442 INFO:teuthology.task.ansible.out: "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-07-23 12:21:50 UTC", "StateChangeTimestampMonotonic": "20948857", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-07-23 12:21:50 UTC", "WatchdogTimestampMonotonic": "20948856", "WatchdogUSec": "0"}} 2020-07-23T12:23:59.450 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:59 +0000 (0:00:00.244) 0:01:59.281 ********* 2020-07-23T12:23:59.494 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-07-23T12:23:59.496 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:59.551 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:59 +0000 (0:00:00.100) 0:01:59.381 ********* 2020-07-23T12:23:59.553 INFO:teuthology.task.ansible.out: 2020-07-23T12:23:59.568 INFO:teuthology.task.ansible.out: 2020-07-23T12:23:59.570 INFO:teuthology.task.ansible.out:TASK [testnode : Stop apache2] ************************************************* 2020-07-23T12:23:59.571 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2020-07-23 12:21:47 UTC", "ActiveEnterTimestampMonotonic": "21490234", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket system.slice sysinit.target remote-fs.target basic.target -.mount network.target nss-lookup.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-07-23 12:21:46 UTC", "AssertTimestampMonotonic": "21126642", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-07-23 12:21:46 UTC", "ConditionTimestampMonotonic": "21126642", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-07-23T12:23:59.573 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1515", "ExecMainStartTimestamp": "Thu 2020-07-23 12:21:47 UTC", "ExecMainStartTimestampMonotonic": "21490217", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-07-23 12:21:46 UTC", "InactiveExitTimestampMonotonic": "21128268", "InvocationID": "a777c84abaad452f9e4b98449ae80efe", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-07-23T12:23:59.575 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1515", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-07-23T12:23:59.576 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-07-23 12:21:47 UTC", "StateChangeTimestampMonotonic": "21490234", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-07-23 12:21:47 UTC", "WatchdogTimestampMonotonic": "21490233", "WatchdogUSec": "0"}} 2020-07-23T12:23:59.625 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:59 +0000 (0:00:00.074) 0:01:59.455 ********* 2020-07-23T12:23:59.635 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-07-23T12:23:59.637 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:59.678 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:59 +0000 (0:00:00.052) 0:01:59.508 ********* 2020-07-23T12:23:59.695 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:59.736 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:59 +0000 (0:00:00.056) 0:01:59.565 ********* 2020-07-23T12:23:59.746 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-07-23T12:23:59.748 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:59.789 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:59 +0000 (0:00:00.053) 0:01:59.619 ********* 2020-07-23T12:23:59.800 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:59.864 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:59 +0000 (0:00:00.075) 0:01:59.694 ********* 2020-07-23T12:23:59.876 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-07-23T12:23:59.878 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:59.921 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:59 +0000 (0:00:00.056) 0:01:59.751 ********* 2020-07-23T12:23:59.931 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-07-23T12:23:59.933 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:23:59.976 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:23:59 +0000 (0:00:00.055) 0:01:59.806 ********* 2020-07-23T12:23:59.988 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-07-23T12:23:59.990 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.033 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.057) 0:01:59.863 ********* 2020-07-23T12:24:00.044 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-07-23T12:24:00.046 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.085 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.051) 0:01:59.915 ********* 2020-07-23T12:24:00.094 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-07-23T12:24:00.096 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.136 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.050) 0:01:59.966 ********* 2020-07-23T12:24:00.146 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-07-23T12:24:00.148 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.192 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.056) 0:02:00.022 ********* 2020-07-23T12:24:00.206 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.250 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.058) 0:02:00.080 ********* 2020-07-23T12:24:00.260 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-07-23T12:24:00.262 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.300 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.049) 0:02:00.130 ********* 2020-07-23T12:24:00.309 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-07-23T12:24:00.311 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.361 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.061) 0:02:00.191 ********* 2020-07-23T12:24:00.375 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-07-23T12:24:00.376 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.413 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.052) 0:02:00.243 ********* 2020-07-23T12:24:00.422 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-23T12:24:00.424 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.462 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.048) 0:02:00.292 ********* 2020-07-23T12:24:00.472 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-07-23T12:24:00.474 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.514 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.052) 0:02:00.344 ********* 2020-07-23T12:24:00.524 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-07-23T12:24:00.526 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.566 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.049) 0:02:00.393 ********* 2020-07-23T12:24:00.575 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-23T12:24:00.577 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.618 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.054) 0:02:00.448 ********* 2020-07-23T12:24:00.637 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-07-23T12:24:00.640 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.684 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.065) 0:02:00.514 ********* 2020-07-23T12:24:00.695 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-07-23T12:24:00.697 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.736 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.051) 0:02:00.565 ********* 2020-07-23T12:24:00.766 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.787 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.051) 0:02:00.617 ********* 2020-07-23T12:24:00.797 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-07-23T12:24:00.799 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.838 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.050) 0:02:00.667 ********* 2020-07-23T12:24:00.846 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-07-23T12:24:00.848 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.893 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.056) 0:02:00.724 ********* 2020-07-23T12:24:00.903 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-07-23T12:24:00.905 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:00.948 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:00 +0000 (0:00:00.050) 0:02:00.774 ********* 2020-07-23T12:24:00.954 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-07-23T12:24:00.955 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:01.008 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:01 +0000 (0:00:00.063) 0:02:00.838 ********* 2020-07-23T12:24:01.018 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-07-23T12:24:01.020 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:01.070 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:01 +0000 (0:00:00.062) 0:02:00.900 ********* 2020-07-23T12:24:01.101 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-07-23T12:24:01.103 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:01.145 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:01 +0000 (0:00:00.074) 0:02:00.974 ********* 2020-07-23T12:24:01.156 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:01.217 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:01 +0000 (0:00:00.070) 0:02:01.045 ********* 2020-07-23T12:24:01.226 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-07-23T12:24:01.227 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:01.293 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:01 +0000 (0:00:00.077) 0:02:01.123 ********* 2020-07-23T12:24:01.312 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:01.351 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:01 +0000 (0:00:00.057) 0:02:01.180 ********* 2020-07-23T12:24:01.360 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-07-23T12:24:01.362 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:01.401 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:01 +0000 (0:00:00.049) 0:02:01.230 ********* 2020-07-23T12:24:01.411 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-07-23T12:24:01.416 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:01.464 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:01 +0000 (0:00:00.063) 0:02:01.294 ********* 2020-07-23T12:24:01.478 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-07-23T12:24:01.480 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:01.532 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:01 +0000 (0:00:00.068) 0:02:01.362 ********* 2020-07-23T12:24:01.544 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-07-23T12:24:01.546 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:01.586 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:01 +0000 (0:00:00.053) 0:02:01.415 ********* 2020-07-23T12:24:01.602 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:01.643 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:01 +0000 (0:00:00.057) 0:02:01.472 ********* 2020-07-23T12:24:01.653 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-07-23T12:24:01.657 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:01.699 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:01 +0000 (0:00:00.057) 0:02:01.530 ********* 2020-07-23T12:24:01.710 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:01.750 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:01 +0000 (0:00:00.050) 0:02:01.580 ********* 2020-07-23T12:24:01.760 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-07-23T12:24:01.763 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:01.830 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:01 +0000 (0:00:00.053) 0:02:01.634 ********* TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:01.855 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:01 +0000 (0:00:00.048) 0:02:01.682 ********* 2020-07-23T12:24:01.862 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-07-23T12:24:01.866 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:01.902 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:01 +0000 (0:00:00.049) 0:02:01.732 ********* 2020-07-23T12:24:01.912 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-07-23T12:24:01.915 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:01.961 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:01 +0000 (0:00:00.058) 0:02:01.791 ********* 2020-07-23T12:24:01.978 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:02.014 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:02 +0000 (0:00:00.053) 0:02:01.844 ********* 2020-07-23T12:24:02.031 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:02.075 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:02 +0000 (0:00:00.060) 0:02:01.905 ********* 2020-07-23T12:24:02.087 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-07-23T12:24:02.089 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:02.127 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:02 +0000 (0:00:00.052) 0:02:01.957 ********* 2020-07-23T12:24:02.138 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:02.181 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:02 +0000 (0:00:00.053) 0:02:02.011 ********* 2020-07-23T12:24:02.191 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-07-23T12:24:02.193 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:02.229 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:02 +0000 (0:00:00.048) 0:02:02.059 ********* 2020-07-23T12:24:02.242 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:02.290 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:02 +0000 (0:00:00.060) 0:02:02.120 ********* 2020-07-23T12:24:02.300 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-07-23T12:24:02.302 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:02.341 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:02 +0000 (0:00:00.050) 0:02:02.171 ********* 2020-07-23T12:24:02.353 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-07-23T12:24:02.355 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:02.393 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:02 +0000 (0:00:00.051) 0:02:02.223 ********* 2020-07-23T12:24:02.415 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:02.446 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:02 +0000 (0:00:00.052) 0:02:02.276 ********* 2020-07-23T12:24:02.488 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:02.527 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:02 +0000 (0:00:00.081) 0:02:02.357 ********* 2020-07-23T12:24:02.562 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-07-23T12:24:02.564 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-07-23T12:24:02.607 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:02 +0000 (0:00:00.079) 0:02:02.437 ********* 2020-07-23T12:24:02.684 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-07-23T12:24:02.738 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:02 +0000 (0:00:00.130) 0:02:02.567 ********* 2020-07-23T12:24:03.002 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-07-23T12:24:03.049 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:03 +0000 (0:00:00.311) 0:02:02.879 ********* 2020-07-23T12:24:03.104 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-07-23T12:24:03.143 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:03 +0000 (0:00:00.094) 0:02:02.973 ********* 2020-07-23T12:24:03.153 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-07-23T12:24:03.155 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:03.194 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:03 +0000 (0:00:00.050) 0:02:03.023 ********* 2020-07-23T12:24:03.205 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:03.247 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:03 +0000 (0:00:00.053) 0:02:03.077 ********* 2020-07-23T12:24:03.575 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-07-23T12:24:03.614 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:03 +0000 (0:00:00.366) 0:02:03.444 ********* 2020-07-23T12:24:03.733 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-07-23T12:24:03.775 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:03 +0000 (0:00:00.160) 0:02:03.604 ********* 2020-07-23T12:24:04.014 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-07-23T12:24:04.016 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-07-23T12:24:04.055 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:04 +0000 (0:00:00.280) 0:02:03.885 ********* 2020-07-23T12:24:04.184 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-07-23T12:24:04.229 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:04 +0000 (0:00:00.173) 0:02:04.059 ********* 2020-07-23T12:24:04.461 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-07-23T12:24:04.463 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-07-23T12:24:04.508 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:04 +0000 (0:00:00.278) 0:02:04.337 ********* 2020-07-23T12:24:04.630 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-07-23T12:24:04.702 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:04 +0000 (0:00:00.193) 0:02:04.531 ********* 2020-07-23T12:24:04.714 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** ok: [smithi133.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-23T12:24:04.763 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:04 +0000 (0:00:00.062) 0:02:04.593 ********* 2020-07-23T12:24:04.797 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-07-23T12:24:04.798 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:04.838 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:04 +0000 (0:00:00.074) 0:02:04.668 ********* 2020-07-23T12:24:04.862 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-07-23T12:24:04.863 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-23T12:24:04.903 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:04 +0000 (0:00:00.064) 0:02:04.733 ********* 2020-07-23T12:24:04.914 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-07-23T12:24:04.916 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:04.966 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:04 +0000 (0:00:00.061) 0:02:04.794 ********* 2020-07-23T12:24:04.982 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.017 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:05 +0000 (0:00:00.053) 0:02:04.847 ********* 2020-07-23T12:24:05.030 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.079 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:05 +0000 (0:00:00.062) 0:02:04.909 ********* 2020-07-23T12:24:05.102 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-07-23T12:24:05.104 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.142 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:05 +0000 (0:00:00.062) 0:02:04.972 ********* 2020-07-23T12:24:05.154 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.197 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:05 +0000 (0:00:00.055) 0:02:05.027 ********* 2020-07-23T12:24:05.208 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.253 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:05 +0000 (0:00:00.055) 0:02:05.083 ********* 2020-07-23T12:24:05.262 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-07-23T12:24:05.264 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.306 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:05 +0000 (0:00:00.053) 0:02:05.136 ********* 2020-07-23T12:24:05.315 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-07-23T12:24:05.316 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.379 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:05 +0000 (0:00:00.073) 0:02:05.209 ********* 2020-07-23T12:24:05.389 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-07-23T12:24:05.443 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:05 +0000 (0:00:00.050) 0:02:05.260 ********* TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.485 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:05 +0000 (0:00:00.055) 0:02:05.315 ********* 2020-07-23T12:24:05.502 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi133.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.510 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.515 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.527 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.529 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.551 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi137.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-07-23T12:24:05.553 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** ok: [smithi133.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-07-23T12:24:05.598 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:05 +0000 (0:00:00.112) 0:02:05.427 ********* 2020-07-23T12:24:05.609 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.611 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.624 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.625 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.626 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.688 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:05 +0000 (0:00:00.090) 0:02:05.518 ********* 2020-07-23T12:24:05.702 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.703 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.704 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.704 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.705 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.716 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.728 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-07-23T12:24:05.744 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.746 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.787 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.797 INFO:teuthology.task.ansible.out: skipping: [smithi137.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.798 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.799 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.799 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:05.853 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:05 +0000 (0:00:00.165) 0:02:05.683 ********* 2020-07-23T12:24:05.966 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi133.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.009681", "end": "2020-07-23 12:24:05.912950", "rc": 0, "start": "2020-07-23 12:24:05.903269", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-07-23T12:24:06.007 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:06 +0000 (0:00:00.154) 0:02:05.837 ********* 2020-07-23T12:24:06.060 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:06.112 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:06 +0000 (0:00:00.104) 0:02:05.942 ********* 2020-07-23T12:24:06.122 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi137.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004474", "end": "2020-07-23 12:24:06.082230", "rc": 0, "start": "2020-07-23 12:24:06.077756", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-07-23T12:24:06.198 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:06 +0000 (0:00:00.086) 0:02:06.028 ********* 2020-07-23T12:24:06.209 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:06.254 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:06 +0000 (0:00:00.056) 0:02:06.084 ********* 2020-07-23T12:24:06.265 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-07-23T12:24:06.267 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:06.310 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:06 +0000 (0:00:00.055) 0:02:06.140 ********* 2020-07-23T12:24:06.320 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-07-23T12:24:06.322 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:06.361 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:06 +0000 (0:00:00.050) 0:02:06.191 ********* 2020-07-23T12:24:06.372 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-07-23T12:24:06.373 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:06.417 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:06 +0000 (0:00:00.054) 0:02:06.246 ********* 2020-07-23T12:24:06.450 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:06.490 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:06 +0000 (0:00:00.074) 0:02:06.320 ********* 2020-07-23T12:24:07.266 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-07-23T12:24:07.268 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1595506758, "cache_updated": false, "changed": false} 2020-07-23T12:24:07.314 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:07 +0000 (0:00:00.822) 0:02:07.143 ********* 2020-07-23T12:24:07.346 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-07-23T12:24:07.348 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:07.394 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:07 +0000 (0:00:00.080) 0:02:07.224 ********* 2020-07-23T12:24:07.463 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi133.front.sepia.ceph.com] => (item={'block_used': 13745202, 'uuid': 'c4409019-95e1-4faa-8278-2a840007765d', 'size_total': 983416987648, 'block_total': 240092038, 'mount': '/', 'block_available': 226346836, 'size_available': 927116640256, 'fstype': 'ext4', 'inode_total': 61046784, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149409, 'block_size': 4096, 'inode_available': 60897375}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226346836, "block_size": 4096, "block_total": 240092038, "block_used": 13745202, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60897375, "inode_total": 61046784, "inode_used": 149409, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927116640256, "size_total": 983416987648, "uuid": "c4409019-95e1-4faa-8278-2a840007765d"}} 2020-07-23T12:24:07.511 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:07 +0000 (0:00:00.117) 0:02:07.341 ********* 2020-07-23T12:24:07.539 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* ok: [smithi137.front.sepia.ceph.com] => {"cache_update_time": 1595506715, "cache_updated": false, "changed": false} 2020-07-23T12:24:07.585 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:07 +0000 (0:00:00.073) 0:02:07.415 ********* 2020-07-23T12:24:07.624 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:07.666 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:07 +0000 (0:00:00.080) 0:02:07.496 ********* 2020-07-23T12:24:07.709 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-07-23T12:24:07.713 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.010992", "end": "2020-07-23 12:24:07.793163", "rc": 0, "start": "2020-07-23 12:24:07.782171", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:07.765 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:07 +0000 (0:00:00.098) 0:02:07.594 ********* 2020-07-23T12:24:07.776 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-07-23T12:24:07.778 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item={'block_used': 13745203, 'uuid': 'c4409019-95e1-4faa-8278-2a840007765d', 'size_total': 983416987648, 'block_total': 240092038, 'mount': '/', 'block_available': 226346835, 'size_available': 927116636160, 'fstype': 'ext4', 'inode_total': 61046784, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149409, 'block_size': 4096, 'inode_available': 60897375}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226346835, "block_size": 4096, "block_total": 240092038, "block_used": 13745203, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60897375, "inode_total": 61046784, "inode_used": 149409, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927116636160, "size_total": 983416987648, "uuid": "c4409019-95e1-4faa-8278-2a840007765d"}} 2020-07-23T12:24:07.823 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:07 +0000 (0:00:00.059) 0:02:07.654 ********* 2020-07-23T12:24:07.833 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-07-23T12:24:07.835 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'block_used': 13745202, 'uuid': 'c4409019-95e1-4faa-8278-2a840007765d', 'size_total': 983416987648, 'block_total': 240092038, 'mount': '/', 'block_available': 226346836, 'size_available': 927116640256, 'fstype': 'ext4', 'inode_total': 61046784, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149409, 'block_size': 4096, 'inode_available': 60897375}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226346836, "block_size": 4096, "block_total": 240092038, "block_used": 13745202, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60897375, "inode_total": 61046784, "inode_used": 149409, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927116640256, "size_total": 983416987648, "uuid": "c4409019-95e1-4faa-8278-2a840007765d"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:07.880 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:07 +0000 (0:00:00.056) 0:02:07.710 ********* 2020-07-23T12:24:07.913 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-07-23T12:24:07.955 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:07 +0000 (0:00:00.074) 0:02:07.785 ********* 2020-07-23T12:24:07.977 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-07-23T12:24:07.979 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009848", "end": "2020-07-23 12:24:08.003056", "rc": 0, "start": "2020-07-23 12:24:07.993208", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:08.028 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:08 +0000 (0:00:00.072) 0:02:07.858 ********* 2020-07-23T12:24:08.040 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:08.086 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-07-23T12:24:08.088 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'block_used': 13745203, 'uuid': 'c4409019-95e1-4faa-8278-2a840007765d', 'size_total': 983416987648, 'block_total': 240092038, 'mount': '/', 'block_available': 226346835, 'size_available': 927116636160, 'fstype': 'ext4', 'inode_total': 61046784, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149409, 'block_size': 4096, 'inode_available': 60897375}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226346835, "block_size": 4096, "block_total": 240092038, "block_used": 13745203, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60897375, "inode_total": 61046784, "inode_used": 149409, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927116636160, "size_total": 983416987648, "uuid": "c4409019-95e1-4faa-8278-2a840007765d"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:08.130 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:08 +0000 (0:00:00.101) 0:02:07.960 ********* 2020-07-23T12:24:08.173 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-07-23T12:24:08.233 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:08 +0000 (0:00:00.103) 0:02:08.063 ********* 2020-07-23T12:24:08.297 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:10.001 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgOc9Pz0XjfCe0ZfQVPInZ5HLVYqW5YDK1p'], 'uuids': ['b8876328-ebb3-4fa0-99e5-1f224bb4a7d6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.577129", "end": "2020-07-23 12:24:10.020278", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgOc9Pz0XjfCe0ZfQVPInZ5HLVYqW5YDK1p"], "labels": [], "masters": [], "uuids": ["b8876328-ebb3-4fa0-99e5-1f224bb4a7d6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:08.443149", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-23T12:24:10.013 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:10.026 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:10.114 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl7q1HrPdvyjfhSpkXCQR3qCdIVeR1WfADW'], 'uuids': ['aec362cb-920f-4910-ba60-f98993371c6e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.950763", "end": "2020-07-23 12:24:10.185550", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl7q1HrPdvyjfhSpkXCQR3qCdIVeR1WfADW"], "labels": [], "masters": [], "uuids": ["aec362cb-920f-4910-ba60-f98993371c6e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:08.234787", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-23T12:24:10.124 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:10.146 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:11.345 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgO5Uqy0zo4wYc4KzWXv5Kxr7GGuvc4QEeG'], 'uuids': ['17511dd4-a38f-4433-b876-b5a7310a11bd']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.166838", "end": "2020-07-23 12:24:11.349031", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgO5Uqy0zo4wYc4KzWXv5Kxr7GGuvc4QEeG"], "labels": [], "masters": [], "uuids": ["17511dd4-a38f-4433-b876-b5a7310a11bd"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:10.182193", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi137.front.sepia.ceph.com] => 2020-07-23T12:24:11.347 INFO:teuthology.task.ansible.out:(item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLAR', 'wwn-0x5000c5009293fb5d'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c4409019-95e1-4faa-8278-2a840007765d', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLAR-part1', 'wwn-0x5000c5009293fb5d-part1'], 'uuids': ['c4409019-95e1-4faa-8278-2a840007765d']}, 'sectors': '1953494016', 'start': '2048', 'holders': [], 'size': '931.50 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009293fb5d', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XLAR', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLAR", "wwn-0x5000c5009293fb5d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLAR-part1", "wwn-0x5000c5009293fb5d-part1"], "labels": [], "masters": [], "uuids": ["c4409019-95e1-4faa-8278-2a840007765d"]}, "sectors": "1953494016", "sectorsize": 512, "size": "931.50 GB", "start": "2048", "uuid": "c4409019-95e1-4faa-8278-2a840007765d"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XLAR", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293fb5d"}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:11.430 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl7XExZp2YRHH2FAybTDiCtmN4KZ9HZkyAQ'], 'uuids': ['a9cd036f-93bb-4c40-a463-a3e7ef795894']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.138088", "end": "2020-07-23 12:24:11.510139", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl7XExZp2YRHH2FAybTDiCtmN4KZ9HZkyAQ"], "labels": [], "masters": [], "uuids": ["a9cd036f-93bb-4c40-a463-a3e7ef795894"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:10.372051", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-23T12:24:11.443 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XPD9', 'wwn-0x5000c50092928ee8'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c4409019-95e1-4faa-8278-2a840007765d', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XPD9-part1', 'wwn-0x5000c50092928ee8-part1'], 'uuids': ['c4409019-95e1-4faa-8278-2a840007765d']}, 'sectors': '1953494016', 'start': '2048', 'holders': [], 'size': '931.50 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092928ee8', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XPD9', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XPD9", "wwn-0x5000c50092928ee8"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XPD9-part1", "wwn-0x5000c50092928ee8-part1"], "labels": [], "masters": [], "uuids": ["c4409019-95e1-4faa-8278-2a840007765d"]}, "sectors": "1953494016", "sectorsize": 512, "size": "931.50 GB", "start": "2048", "uuid": "c4409019-95e1-4faa-8278-2a840007765d"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XPD9", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092928ee8"}}, "skip_reason": 2020-07-23T12:24:11.445 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-07-23T12:24:12.632 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-0jKIIH-n7ME-NE4G-7l4B-mplr-SetZ-voocwI', 'nvme-INTEL_SSDPEDMD400G4_CVFT6233004A400BGN', 'nvme-nvme.8086-43564654363233333030344134303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.141645", "end": "2020-07-23 12:24:12.640282", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-0jKIIH-n7ME-NE4G-7l4B-mplr-SetZ-voocwI", "nvme-INTEL_SSDPEDMD400G4_CVFT6233004A400BGN", "nvme-nvme.8086-43564654363233333030344134303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:11.498637", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-07-23T12:24:12.636 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-23T12:24:12.644 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:12.650 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:12.816 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-fG1Tgg-vt2I-Ywzn-oFa1-LL3b-R2ws-bFecEq', 'nvme-INTEL_SSDPEDMD400G4_PHFT620401AQ400BGN', 'nvme-nvme.8086-50484654363230343031415134303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.200256", "end": "2020-07-23 12:24:12.880241", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-fG1Tgg-vt2I-Ywzn-oFa1-LL3b-R2ws-bFecEq", "nvme-INTEL_SSDPEDMD400G4_PHFT620401AQ400BGN", "nvme-nvme.8086-50484654363230343031415134303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:11.679985", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-07-23T12:24:12.818 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-23T12:24:12.821 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:12.832 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:14.060 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgOU4O7VsylLanQdyLNV6kpMKadkn9klOcN'], 'uuids': ['7ba89931-0458-409d-8d70-acc4b7a15388']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.285626", "end": "2020-07-23 12:24:14.076490", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgOU4O7VsylLanQdyLNV6kpMKadkn9klOcN"], "labels": [], "masters": [], "uuids": ["7ba89931-0458-409d-8d70-acc4b7a15388"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:12.790864", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-23T12:24:14.085 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:14.132 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl7ZyJZlFnRgkW3ShH7Zxk6a9SjtMcxqzpD'], 'uuids': ['7bcff275-deb9-4351-a39b-8d2563a19342']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.141553", "end": "2020-07-23 12:24:14.200827", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl7ZyJZlFnRgkW3ShH7Zxk6a9SjtMcxqzpD"], "labels": [], "masters": [], "uuids": ["7bcff275-deb9-4351-a39b-8d2563a19342"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:13.059274", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-23T12:24:14.150 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:14.152 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:15.443 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl7UE0zGcnO9ZavtQE0kjdUkFFu7tGSTB7H'], 'uuids': ['e7bc00e9-0d7e-48b9-a85a-ba9c86d457a3']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.152799", "end": "2020-07-23 12:24:15.525378", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl7UE0zGcnO9ZavtQE0kjdUkFFu7tGSTB7H"], "labels": [], "masters": [], "uuids": ["e7bc00e9-0d7e-48b9-a85a-ba9c86d457a3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:14.372579", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-23T12:24:15.455 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:15.535 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgO4Pg59oxwnoQWYf4Oyn2LcNKU7VXcPDcn'], 'uuids': ['563c965a-2bec-42d6-beb7-316303f79d7a']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.293914", "end": "2020-07-23 12:24:15.548479", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgO4Pg59oxwnoQWYf4Oyn2LcNKU7VXcPDcn"], "labels": [], "masters": [], "uuids": ["563c965a-2bec-42d6-beb7-316303f79d7a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:14.254565", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-23T12:24:15.549 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:16.728 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl74XeTEscsYabfMAu06EYgwRPXPKfqVQfw'], 'uuids': ['7ccf0843-7885-4433-93f3-6341402acd04']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.146053", "end": "2020-07-23 12:24:16.808029", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl74XeTEscsYabfMAu06EYgwRPXPKfqVQfw"], "labels": [], "masters": [], "uuids": ["7ccf0843-7885-4433-93f3-6341402acd04"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:15.661976", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-07-23T12:24:16.796 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:16 +0000 (0:00:08.559) 0:02:16.623 ********* 2020-07-23T12:24:16.857 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:17.039 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgO1ah4Yevxx1peQCIknQILhh8hWmNxyC0N'], 'uuids': ['e9c197a4-9921-4727-b871-05ca21c44c73']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.366708", "end": "2020-07-23 12:24:17.061145", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgO1ah4Yevxx1peQCIknQILhh8hWmNxyC0N"], "labels": [], "masters": [], "uuids": ["e9c197a4-9921-4727-b871-05ca21c44c73"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:15.694437", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk u 2020-07-23T12:24:17.041 INFO:teuthology.task.ansible.out:sing fdisk or", "other utilities."]} 2020-07-23T12:24:17.098 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:17 +0000 (0:00:00.305) 0:02:16.928 ********* 2020-07-23T12:24:17.158 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-07-23T12:24:17.160 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl7q1HrPdvyjfhSpkXCQR3qCdIVeR1WfADW'], 'uuids': ['aec362cb-920f-4910-ba60-f98993371c6e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.154927", "end": "2020-07-23 12:24:17.226228", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl7q1HrPdvyjfhSpkXCQR3qCdIVeR1WfADW"], "labels": [], "masters": [], "uuids": ["aec362cb-920f-4910-ba60-f98993371c6e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:17.071301", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138748 s, 831 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138748 s, 831 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:17.170 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:17.171 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:17.173 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:17.456 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl7XExZp2YRHH2FAybTDiCtmN4KZ9HZkyAQ'], 'uuids': ['a9cd036f-93bb-4c40-a463-a3e7ef795894']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.144918", "end": "2020-07-23 12:24:17.529913", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl7XExZp2YRHH2FAybTDiCtmN4KZ9HZkyAQ"], "labels": [], "masters": [], "uuids": ["a9cd036f-93bb-4c40-a463-a3e7ef795894"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:17.384995", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140893 s, 819 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140893 s, 819 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:17.465 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XPD9', 'wwn-0x5000c50092928ee8'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c4409019-95e1-4faa-8278-2a840007765d', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XPD9-part1', 'wwn-0x5000c50092928ee8-part1'], 'uuids': ['c4409019-95e1-4faa-8278-2a840007765d']}, 'sectors': '1953494016', 'start': '2048', 'holders': [], 'size': '931.50 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092928ee8', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XPD9', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XPD9", "wwn-0x5000c50092928ee8"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XPD9-part1", "wwn-0x5000c50092928ee8-part1"], "labels": [], "masters": [], "uuids": ["c4409019-95e1-4faa-8278-2a840007765d"]}, "sectors": "1953494016", "sectorsize": 512, "size": "931.50 GB", "start": "2048", "uuid": "c4409019-95e1-4faa-8278-2a840007765d"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XPD9", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092928ee8"}}, "skip_reason": 2020-07-23T12:24:17.467 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-07-23T12:24:17.551 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgOc9Pz0XjfCe0ZfQVPInZ5HLVYqW5YDK1p'], 'uuids': ['b8876328-ebb3-4fa0-99e5-1f224bb4a7d6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.253312", "end": "2020-07-23 12:24:17.566727", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgOc9Pz0XjfCe0ZfQVPInZ5HLVYqW5YDK1p"], "labels": [], "masters": [], "uuids": ["b8876328-ebb3-4fa0-99e5-1f224bb4a7d6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:17.313415", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139548 s, 827 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139548 s, 827 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:17.584 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:17.587 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:17.662 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-fG1Tgg-vt2I-Ywzn-oFa1-LL3b-R2ws-bFecEq', 'nvme-INTEL_SSDPEDMD400G4_PHFT620401AQ400BGN', 'nvme-nvme.8086-50484654363230343031415134303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.049440", "end": "2020-07-23 12:24:17.736399", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-fG1Tgg-vt2I-Ywzn-oFa1-LL3b-R2ws-bFecEq", "nvme-INTEL_SSDPEDMD400G4_PHFT620401AQ400BGN", "nvme-nvme.8086-50484654363230343031415134303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:17.686959", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0447468 s, 2.6 GB/s", "std 2020-07-23T12:24:17.664 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0447468 s, 2.6 GB/s"], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:17.669 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:17.703 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:17.873 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgO5Uqy0zo4wYc4KzWXv5Kxr7GGuvc4QEeG'], 'uuids': ['17511dd4-a38f-4433-b876-b5a7310a11bd']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.152103", "end": "2020-07-23 12:24:17.885041", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgO5Uqy0zo4wYc4KzWXv5Kxr7GGuvc4QEeG"], "labels": [], "masters": [], "uuids": ["17511dd4-a38f-4433-b876-b5a7310a11bd"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:17.732938", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.147458 s, 782 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.147458 s, 782 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:17.876 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLAR', 'wwn-0x5000c5009293fb5d'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c4409019-95e1-4faa-8278-2a840007765d', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLAR-part1', 'wwn-0x5000c5009293fb5d-part1'], 'uuids': ['c4409019-95e1-4faa-8278-2a840007765d']}, 'sectors': '1953494016', 'start': '2048', 'holders': [], 'size': '931.50 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009293fb5d', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XLAR', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLAR", "wwn-0x5000c5009293fb5d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLAR-part1", "wwn-0x5000c5009293fb5d-part1"], "labels": [], "masters": [], "uuids": ["c4409019-95e1-4faa-8278-2a840007765d"]}, "sectors": "1953494016", "sectorsize": 512, "size": "931.50 GB", "start": "2048", "uuid": "c4409019-95e1-4faa-8278-2a840007765d"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XLAR", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293fb5d"}}, "skip_reason": 2020-07-23T12:24:17.878 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-07-23T12:24:17.991 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl7ZyJZlFnRgkW3ShH7Zxk6a9SjtMcxqzpD'], 'uuids': ['7bcff275-deb9-4351-a39b-8d2563a19342']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.147546", "end": "2020-07-23 12:24:18.062911", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl7ZyJZlFnRgkW3ShH7Zxk6a9SjtMcxqzpD"], "labels": [], "masters": [], "uuids": ["7bcff275-deb9-4351-a39b-8d2563a19342"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:17.915365", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144608 s, 798 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144608 s, 798 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:18.005 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:18.026 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:18.098 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-0jKIIH-n7ME-NE4G-7l4B-mplr-SetZ-voocwI', 'nvme-INTEL_SSDPEDMD400G4_CVFT6233004A400BGN', 'nvme-nvme.8086-43564654363233333030344134303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.049283", "end": "2020-07-23 12:24:18.094619", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-0jKIIH-n7ME-NE4G-7l4B-mplr-SetZ-voocwI", "nvme-INTEL_SSDPEDMD400G4_CVFT6233004A400BGN", "nvme-nvme.8086-43564654363233333030344134303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:18.045336", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0447213 s, 2.6 GB/s", "std 2020-07-23T12:24:18.100 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0447213 s, 2.6 GB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:18.110 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:18.355 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl7UE0zGcnO9ZavtQE0kjdUkFFu7tGSTB7H'], 'uuids': ['e7bc00e9-0d7e-48b9-a85a-ba9c86d457a3']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.160367", "end": "2020-07-23 12:24:18.405660", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl7UE0zGcnO9ZavtQE0kjdUkFFu7tGSTB7H"], "labels": [], "masters": [], "uuids": ["e7bc00e9-0d7e-48b9-a85a-ba9c86d457a3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:18.245293", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.155649 s, 741 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.155649 s, 741 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:18.367 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:18.381 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgOU4O7VsylLanQdyLNV6kpMKadkn9klOcN'], 'uuids': ['7ba89931-0458-409d-8d70-acc4b7a15388']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.154092", "end": "2020-07-23 12:24:18.395050", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgOU4O7VsylLanQdyLNV6kpMKadkn9klOcN"], "labels": [], "masters": [], "uuids": ["7ba89931-0458-409d-8d70-acc4b7a15388"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:18.240958", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151347 s, 762 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151347 s, 762 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:18.426 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:18.687 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl74XeTEscsYabfMAu06EYgwRPXPKfqVQfw'], 'uuids': ['7ccf0843-7885-4433-93f3-6341402acd04']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.157295", "end": "2020-07-23 12:24:18.758425", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-c1hNOD4YUbVOfbFeS5LH0uidge3Pycl74XeTEscsYabfMAu06EYgwRPXPKfqVQfw"], "labels": [], "masters": [], "uuids": ["7ccf0843-7885-4433-93f3-6341402acd04"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:18.601130", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153884 s, 750 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153884 s, 750 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:18.744 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:18 +0000 (0:00:01.645) 0:02:18.574 ********* 2020-07-23T12:24:18.764 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgO4Pg59oxwnoQWYf4Oyn2LcNKU7VXcPDcn'], 'uuids': ['563c965a-2bec-42d6-beb7-316303f79d7a']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.159970", "end": "2020-07-23 12:24:18.729148", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgO4Pg59oxwnoQWYf4Oyn2LcNKU7VXcPDcn"], "labels": [], "masters": [], "uuids": ["563c965a-2bec-42d6-beb7-316303f79d7a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:18.569178", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.155168 s, 743 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.155168 s, 743 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:18.770 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:19.038 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi133.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.110487", "end": "2020-07-23 12:24:19.100050", "rc": 0, "start": "2020-07-23 12:24:18.989563", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgO1ah4Yevxx1peQCIknQILhh8hWmNxyC0N'], 'uuids': ['e9c197a4-9921-4727-b871-05ca21c44c73']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.141647", "end": "2020-07-23 12:24:19.033703", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-V9jxYUG9GrOp7oUTJOybVsEUfRW8GzgO1ah4Yevxx1peQCIknQILhh8hWmNxyC0N"], "labels": [], "masters": [], "uuids": ["e9c197a4-9921-4727-b871-05ca21c44c73"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-07-23 12:24:18.892056", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139332 s, 828 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.13933 2020-07-23T12:24:19.040 INFO:teuthology.task.ansible.out:2 s, 828 MB/s"], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:19.087 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:19 +0000 (0:00:00.342) 0:02:18.917 ********* 2020-07-23T12:24:19.130 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:19 +0000 (0:00:00.042) 0:02:18.960 ********* 2020-07-23T12:24:19.295 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-07-23T12:24:19.303 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.051591", "end": "2020-07-23 12:24:19.382145", "rc": 0, "start": "2020-07-23 12:24:19.330554", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:19.363 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:19 +0000 (0:00:00.227) 0:02:19.187 ********* 2020-07-23T12:24:19.399 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-07-23T12:24:19.461 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:19 +0000 (0:00:00.099) 0:02:19.287 ********* 2020-07-23T12:24:19.476 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi137.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.118231", "end": "2020-07-23 12:24:19.415892", "rc": 0, "start": "2020-07-23 12:24:19.297661", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:19.521 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:19 +0000 (0:00:00.060) 0:02:19.347 ********* 2020-07-23T12:24:19.528 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-07-23T12:24:19.534 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:19.574 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:19 +0000 (0:00:00.052) 0:02:19.400 ********* 2020-07-23T12:24:19.612 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-07-23T12:24:19.614 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:19.683 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:19 +0000 (0:00:00.112) 0:02:19.513 ********* 2020-07-23T12:24:19.726 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-07-23T12:24:19.728 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:19.766 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:19 +0000 (0:00:00.083) 0:02:19.597 ********* 2020-07-23T12:24:19.778 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** changed: [smithi137.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.048289", "end": "2020-07-23 12:24:19.746406", "rc": 0, "start": "2020-07-23 12:24:19.698117", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:19.818 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:19 +0000 (0:00:00.051) 0:02:19.648 ********* 2020-07-23T12:24:19.836 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:19.869 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:19 +0000 (0:00:00.050) 0:02:19.699 ********* 2020-07-23T12:24:19.886 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-07-23T12:24:19.932 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:19 +0000 (0:00:00.062) 0:02:19.762 ********* 2020-07-23T12:24:19.978 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:20.014 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:20 +0000 (0:00:00.082) 0:02:19.844 ********* 2020-07-23T12:24:20.037 INFO:teuthology.task.ansible.out: 2020-07-23T12:24:20.042 INFO:teuthology.task.ansible.out:TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-07-23T12:24:20.044 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-23T12:24:20.044 INFO:teuthology.task.ansible.out: 2020-07-23T12:24:20.103 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:20 +0000 (0:00:00.079) 0:02:19.924 ********* 2020-07-23T12:24:20.106 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:20.166 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:20 +0000 (0:00:00.068) 0:02:19.993 ********* 2020-07-23T12:24:20.178 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi133.front.sepia.ceph.com] => (item={'block_used': 13745202, 'uuid': 'c4409019-95e1-4faa-8278-2a840007765d', 'size_total': 983416987648, 'block_total': 240092038, 'mount': '/', 'block_available': 226346836, 'size_available': 927116640256, 'fstype': 'ext4', 'inode_total': 61046784, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149409, 'block_size': 4096, 'inode_available': 60897375}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226346836, "block_size": 4096, "block_total": 240092038, "block_used": 13745202, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60897375, "inode_total": 61046784, "inode_used": 149409, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927116640256, "size_total": 983416987648, "uuid": "c4409019-95e1-4faa-8278-2a840007765d"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:20.225 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:20 +0000 (0:00:00.062) 0:02:20.055 ********* 2020-07-23T12:24:20.236 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-07-23T12:24:20.238 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:20.282 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:20 +0000 (0:00:00.056) 0:02:20.112 ********* 2020-07-23T12:24:20.303 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:20.350 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:20 +0000 (0:00:00.068) 0:02:20.180 ********* 2020-07-23T12:24:20.361 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-07-23T12:24:20.364 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:20.404 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:20 +0000 (0:00:00.053) 0:02:20.234 ********* 2020-07-23T12:24:20.415 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-23T12:24:20.417 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:20.456 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:20 +0000 (0:00:00.052) 0:02:20.286 ********* 2020-07-23T12:24:20.506 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:20.544 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:20 +0000 (0:00:00.088) 0:02:20.374 ********* 2020-07-23T12:24:20.567 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-07-23T12:24:20.569 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-07-23T12:24:20.608 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:20 +0000 (0:00:00.063) 0:02:20.438 ********* 2020-07-23T12:24:20.618 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-07-23T12:24:20.626 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:20.670 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:20 +0000 (0:00:00.061) 0:02:20.500 ********* 2020-07-23T12:24:20.684 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi137.front.sepia.ceph.com] => (item={'block_used': 13745203, 'uuid': 'c4409019-95e1-4faa-8278-2a840007765d', 'size_total': 983416987648, 'block_total': 240092038, 'mount': '/', 'block_available': 226346835, 'size_available': 927116636160, 'fstype': 'ext4', 'inode_total': 61046784, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149409, 'block_size': 4096, 'inode_available': 60897375}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226346835, "block_size": 4096, "block_total": 240092038, "block_used": 13745203, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60897375, "inode_total": 61046784, "inode_used": 149409, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927116636160, "size_total": 983416987648, "uuid": "c4409019-95e1-4faa-8278-2a840007765d"}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:20.722 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:20 +0000 (0:00:00.052) 0:02:20.552 ********* 2020-07-23T12:24:20.733 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-07-23T12:24:20.734 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:20.792 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:20 +0000 (0:00:00.067) 0:02:20.620 ********* 2020-07-23T12:24:20.801 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-07-23T12:24:20.809 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:20.848 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:20 +0000 (0:00:00.055) 0:02:20.676 ********* 2020-07-23T12:24:20.857 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:20.900 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:20 +0000 (0:00:00.053) 0:02:20.729 ********* 2020-07-23T12:24:20.913 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:20.967 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:20 +0000 (0:00:00.064) 0:02:20.794 ********* 2020-07-23T12:24:21.132 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 23 July 2020 12:24:21 +0000 (0:00:00.091) 0:02:20.886 ********* TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:21.148 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:21 +0000 (0:00:00.091) 0:02:20.978 ********* 2020-07-23T12:24:21.198 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:21.242 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:21 +0000 (0:00:00.086) 0:02:21.065 ********* 2020-07-23T12:24:21.290 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:21.339 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:21 +0000 (0:00:00.102) 0:02:21.168 ********* 2020-07-23T12:24:21.579 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-07-23T12:24:21.581 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-07-23T12:24:21.644 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:21 +0000 (0:00:00.304) 0:02:21.473 ********* 2020-07-23T12:24:21.734 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-07-23T12:24:21.778 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:21 +0000 (0:00:00.135) 0:02:21.608 ********* 2020-07-23T12:24:22.259 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-23T12:24:22.270 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-23T12:24:22.484 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-23T12:24:22.511 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-23T12:24:22.777 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-23T12:24:22.779 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-23T12:24:23.014 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-23T12:24:23.037 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-23T12:24:23.298 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-23T12:24:23.355 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:23 +0000 (0:00:01.577) 0:02:23.185 ********* 2020-07-23T12:24:23.366 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-07-23T12:24:23.428 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:23 +0000 (0:00:00.073) 0:02:23.258 ********* 2020-07-23T12:24:23.522 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-07-23T12:24:23.524 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-07-23T12:24:23.564 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:23 +0000 (0:00:00.135) 0:02:23.394 ********* 2020-07-23T12:24:23.619 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-07-23T12:24:23.672 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:23 +0000 (0:00:00.108) 0:02:23.502 ********* 2020-07-23T12:24:23.740 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-07-23T12:24:23.742 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-23T12:24:23.859 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-23T12:24:23.887 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-23T12:24:24.006 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-23T12:24:24.035 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-23T12:24:24.125 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-23T12:24:24.192 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-23T12:24:24.225 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:24.268 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:24 +0000 (0:00:00.593) 0:02:24.096 ********* 2020-07-23T12:24:24.279 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-07-23T12:24:24.306 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:24.351 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:24 +0000 (0:00:00.085) 0:02:24.181 ********* 2020-07-23T12:24:24.434 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-07-23T12:24:24.441 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-07-23T12:24:24.480 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:24 +0000 (0:00:00.128) 0:02:24.310 ********* 2020-07-23T12:24:24.532 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-07-23T12:24:24.534 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-07-23T12:24:24.579 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:24 +0000 (0:00:00.096) 0:02:24.406 ********* 2020-07-23T12:24:24.633 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:24 +0000 (0:00:00.057) 0:02:24.463 ********* 2020-07-23T12:24:24.708 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-07-23T12:24:24.750 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:24 +0000 (0:00:00.117) 0:02:24.581 ********* 2020-07-23T12:24:25.152 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi133.front.sepia.ceph.com] => {"changed": true} 2020-07-23T12:24:25.223 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:25 +0000 (0:00:00.471) 0:02:25.052 ********* 2020-07-23T12:24:25.236 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => {"changed": true} 2020-07-23T12:24:25.306 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:25 +0000 (0:00:00.071) 0:02:25.123 ********* 2020-07-23T12:24:25.844 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-07-23T12:24:25.846 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} changed: [smithi137.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-07-23T12:24:25.890 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:25 +0000 (0:00:00.591) 0:02:25.715 ********* 2020-07-23T12:24:25.933 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:25 +0000 (0:00:00.048) 0:02:25.763 ********* 2020-07-23T12:24:26.065 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 1224, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-07-23T12:24:26.125 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:26 +0000 (0:00:00.191) 0:02:25.955 ********* 2020-07-23T12:24:26.143 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "gid": 1224, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-07-23T12:24:26.219 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:26 +0000 (0:00:00.093) 0:02:26.049 ********* 2020-07-23T12:24:26.301 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-07-23T12:24:26.347 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:26 +0000 (0:00:00.127) 0:02:26.177 ********* 2020-07-23T12:24:26.411 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-07-23T12:24:26.462 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:26 +0000 (0:00:00.112) 0:02:26.289 ********* 2020-07-23T12:24:26.745 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-07-23T12:24:26.747 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1224, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-07-23T12:24:26.790 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:26 +0000 (0:00:00.331) 0:02:26.621 ********* 2020-07-23T12:24:26.879 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1224, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-07-23T12:24:26.918 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:26 +0000 (0:00:00.127) 0:02:26.748 ********* 2020-07-23T12:24:27.208 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-07-23T12:24:27.210 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-07-23T12:24:27.288 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:27 +0000 (0:00:00.370) 0:02:27.118 ********* 2020-07-23T12:24:27.311 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-07-23T12:24:27.380 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:27 +0000 (0:00:00.081) 0:02:27.199 ********* 2020-07-23T12:24:28.176 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-07-23T12:24:28.179 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1595506758, "cache_updated": false, "changed": false} 2020-07-23T12:24:28.377 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:28 +0000 (0:00:01.008) 0:02:28.207 ********* 2020-07-23T12:24:28.389 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"cache_update_time": 1595506715, "cache_updated": false, "changed": false} 2020-07-23T12:24:28.429 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:28 +0000 (0:00:00.051) 0:02:28.259 ********* 2020-07-23T12:24:28.599 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.047358", "end": "2020-07-23 12:24:28.672777", "rc": 0, "start": "2020-07-23 12:24:28.625419", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-07-23T12:24:28.656 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:28 +0000 (0:00:00.227) 0:02:28.486 ********* 2020-07-23T12:24:28.667 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.046768", "end": "2020-07-23 12:24:28.635691", "rc": 0, "start": "2020-07-23 12:24:28.588923", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-07-23T12:24:28.710 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:28 +0000 (0:00:00.051) 0:02:28.537 ********* 2020-07-23T12:24:28.722 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:28.765 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:28 +0000 (0:00:00.057) 0:02:28.595 ********* 2020-07-23T12:24:28.781 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:28.840 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:28 +0000 (0:00:00.074) 0:02:28.669 ********* 2020-07-23T12:24:28.851 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:28.901 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:28 +0000 (0:00:00.061) 0:02:28.730 ********* 2020-07-23T12:24:28.915 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:28.958 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:28 +0000 (0:00:00.056) 0:02:28.787 ********* 2020-07-23T12:24:29.740 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-07-23T12:24:29.742 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1595506758, "cache_updated": false, "changed": false} 2020-07-23T12:24:29.790 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:29 +0000 (0:00:00.832) 0:02:29.620 ********* 2020-07-23T12:24:29.834 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-07-23T12:24:29.835 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:29.910 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:29 +0000 (0:00:00.120) 0:02:29.740 ********* 2020-07-23T12:24:29.922 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-07-23T12:24:29.924 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"cache_update_time": 1595506715, "cache_updated": false, "changed": false} 2020-07-23T12:24:29.968 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:29 +0000 (0:00:00.057) 0:02:29.798 ********* 2020-07-23T12:24:30.008 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:30.065 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:30 +0000 (0:00:00.097) 0:02:29.895 ********* 2020-07-23T12:24:30.346 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-07-23T12:24:30.348 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-07-23T12:24:30.397 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:30 +0000 (0:00:00.332) 0:02:30.227 ********* 2020-07-23T12:24:30.442 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-07-23T12:24:30.444 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Create the ntp.conf file.] ************************************ ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-07-23T12:24:30.483 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:30 +0000 (0:00:00.084) 0:02:30.312 ********* 2020-07-23T12:24:30.534 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:30 +0000 (0:00:00.050) 0:02:30.363 ********* 2020-07-23T12:24:30.577 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-07-23T12:24:30.579 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:30.617 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:30 +0000 (0:00:00.081) 0:02:30.445 ********* 2020-07-23T12:24:30.750 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-07-23T12:24:30.752 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-07-23 12:21:56 UTC", "ActiveEnterTimestampMonotonic": "26831067", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service network.target basic.target systemd-journald.socket -.mount sysinit.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-07-23 12:21:49 UTC", "AssertTimestampMonotonic": "20176034", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-07-23 12:21:49 UTC", "ConditionTimestampMonotonic": "20176033", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-07-23T12:24:30.754 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1936", "ExecMainStartTimestamp": "Thu 2020-07-23 12:21:56 UTC", "ExecMainStartTimestampMonotonic": "26831051", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-07-23 12:21:49 UTC", "InactiveExitTimestampMonotonic": "20176683", "InvocationID": "a11072b58df9489884a5baa2a4dfac50", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-07-23T12:24:30.756 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1936", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-07-23 12:21:56 UTC", "S 2020-07-23T12:24:30.758 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "26831067", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-07-23 12:21:56 UTC", "WatchdogTimestampMonotonic": "26831062", "WatchdogUSec": "0"}} 2020-07-23T12:24:30.799 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:30 +0000 (0:00:00.184) 0:02:30.630 ********* 2020-07-23T12:24:30.831 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-23T12:24:30.833 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-07-23T12:24:30.910 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:30 +0000 (0:00:00.109) 0:02:30.740 ********* 2020-07-23T12:24:30.922 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-07-23T12:24:30.923 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-07-23 12:21:53 UTC", "ActiveEnterTimestampMonotonic": "27376310", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target systemd-tmpfiles-setup.service systemd-journald.socket system.slice -.mount basic.target network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-07-23 12:21:46 UTC", "AssertTimestampMonotonic": "20744036", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-07-23 12:21:46 UTC", "ConditionTimestampMonotonic": "20744036", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-07-23T12:24:30.924 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1972", "ExecMainStartTimestamp": "Thu 2020-07-23 12:21:53 UTC", "ExecMainStartTimestampMonotonic": "27376298", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-07-23 12:21:46 UTC", "InactiveExitTimestampMonotonic": "20744716", "InvocationID": "73c7e20d412c4eb5845223b01f94ed7e", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128030", "LimitNPROCSoft": "128030", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-07-23T12:24:30.926 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128030", "LimitSIGPENDINGSoft": "128030", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1972", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-07-23 12:21:53 UTC", "S 2020-07-23T12:24:30.928 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "27376310", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-07-23 12:21:53 UTC", "WatchdogTimestampMonotonic": "27376309", "WatchdogUSec": "0"}} 2020-07-23T12:24:30.965 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:30 +0000 (0:00:00.055) 0:02:30.795 ********* 2020-07-23T12:24:30.976 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:31.023 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:31 +0000 (0:00:00.057) 0:02:30.853 ********* 2020-07-23T12:24:31.034 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi137.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-07-23T12:24:31.072 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:31 +0000 (0:00:00.049) 0:02:30.902 ********* 2020-07-23T12:24:31.081 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-07-23T12:24:31.084 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:31.133 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:31 +0000 (0:00:00.053) 0:02:30.956 ********* 2020-07-23T12:24:31.148 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:31.195 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:31 +0000 (0:00:00.066) 0:02:31.024 ********* 2020-07-23T12:24:31.239 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-07-23T12:24:31.240 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:31.279 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:31 +0000 (0:00:00.084) 0:02:31.108 ********* 2020-07-23T12:24:31.990 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-07-23T12:24:31.992 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"cache_update_time": 1595506758, "cache_updated": false, "changed": false} 2020-07-23T12:24:32.033 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:32 +0000 (0:00:00.755) 0:02:31.863 ********* 2020-07-23T12:24:32.077 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-07-23T12:24:32.079 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:32.127 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:32 +0000 (0:00:00.087) 0:02:31.951 ********* 2020-07-23T12:24:32.186 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* ok: [smithi137.front.sepia.ceph.com] => {"cache_update_time": 1595506715, "cache_updated": false, "changed": false} 2020-07-23T12:24:32.246 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:32 +0000 (0:00:00.123) 0:02:32.074 ********* 2020-07-23T12:24:32.277 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-07-23T12:24:32.278 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "gid": 1224, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:32.317 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:32 +0000 (0:00:00.072) 0:02:32.147 ********* 2020-07-23T12:24:32.367 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:32 +0000 (0:00:00.049) 0:02:32.197 ********* 2020-07-23T12:24:32.533 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-07-23T12:24:32.535 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "gid": 1224, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-07-23T12:24:32.598 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:32 +0000 (0:00:00.226) 0:02:32.423 ********* 2020-07-23T12:24:32.813 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1224, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-07-23T12:24:32.865 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:32 +0000 (0:00:00.272) 0:02:32.695 ********* 2020-07-23T12:24:33.049 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1224, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-07-23T12:24:33.148 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:33 +0000 (0:00:00.282) 0:02:32.978 ********* 2020-07-23T12:24:33.963 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-07-23T12:24:34.047 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:34 +0000 (0:00:00.898) 0:02:33.876 ********* 2020-07-23T12:24:34.061 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-07-23T12:24:34.110 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:34 +0000 (0:00:00.063) 0:02:33.940 ********* 2020-07-23T12:24:34.557 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* ok: [smithi133.front.sepia.ceph.com] => {"changed": false} 2020-07-23T12:24:34.624 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:34 +0000 (0:00:00.514) 0:02:34.454 ********* 2020-07-23T12:24:34.666 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-07-23T12:24:34.668 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-07-23T12:24:34.719 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:34 +0000 (0:00:00.091) 0:02:34.545 ********* 2020-07-23T12:24:34.725 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-07-23T12:24:34.727 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false} 2020-07-23T12:24:34.769 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:34 +0000 (0:00:00.053) 0:02:34.599 ********* 2020-07-23T12:24:34.821 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi137.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-07-23T12:24:34.865 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:34 +0000 (0:00:00.095) 0:02:34.695 ********* 2020-07-23T12:24:34.902 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1595506683.3793404, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1595506683.3753405, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 2256, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1595506683.3753405, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "2859737642", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-07-23T12:24:34.946 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:34 +0000 (0:00:00.081) 0:02:34.776 ********* 2020-07-23T12:24:35.057 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1595506693.3795707, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1595506693.3755708, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 2256, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1595506693.3755708, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "2859737642", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-07-23T12:24:35.096 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:35 +0000 (0:00:00.149) 0:02:34.926 ********* 2020-07-23T12:24:35.269 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** ok: [smithi133.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-07-23T12:24:35.322 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:35 +0000 (0:00:00.213) 0:02:35.139 ********* ok: [smithi137.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-07-23T12:24:35.364 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:35 +0000 (0:00:00.054) 0:02:35.194 ********* 2020-07-23T12:24:35.375 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-07-23T12:24:35.377 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-07-23T12:24:35.427 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:35 +0000 (0:00:00.062) 0:02:35.257 ********* 2020-07-23T12:24:35.449 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-07-23T12:24:35.486 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:35 +0000 (0:00:00.059) 0:02:35.316 ********* 2020-07-23T12:24:35.505 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:35.539 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:35 +0000 (0:00:00.052) 0:02:35.369 ********* 2020-07-23T12:24:35.551 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:35.598 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:35 +0000 (0:00:00.055) 0:02:35.425 ********* 2020-07-23T12:24:35.694 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi133.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-07-23T12:24:35.736 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:35 +0000 (0:00:00.140) 0:02:35.566 ********* 2020-07-23T12:24:35.756 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-07-23T12:24:35.799 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:35 +0000 (0:00:00.063) 0:02:35.629 ********* 2020-07-23T12:24:35.903 INFO:teuthology.task.ansible.out: 2020-07-23T12:24:35.905 INFO:teuthology.task.ansible.out:TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi133.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-07-23T12:24:35.963 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:35 +0000 (0:00:00.164) 0:02:35.793 ********* 2020-07-23T12:24:35.986 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-07-23T12:24:36.034 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:36 +0000 (0:00:00.070) 0:02:35.863 ********* 2020-07-23T12:24:36.046 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.094 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:36 +0000 (0:00:00.054) 0:02:35.917 ********* 2020-07-23T12:24:36.101 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.148 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:36 +0000 (0:00:00.061) 0:02:35.978 ********* 2020-07-23T12:24:36.161 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-07-23T12:24:36.163 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.165 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.166 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.208 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:36 +0000 (0:00:00.059) 0:02:36.038 ********* 2020-07-23T12:24:36.219 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.230 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.232 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.291 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:36 +0000 (0:00:00.083) 0:02:36.121 ********* 2020-07-23T12:24:36.304 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi133.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.312 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.318 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi133.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.323 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.368 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:36 +0000 (0:00:00.076) 0:02:36.198 ********* 2020-07-23T12:24:36.380 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.382 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi137.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.382 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.421 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:36 +0000 (0:00:00.052) 0:02:36.251 ********* 2020-07-23T12:24:36.433 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-07-23T12:24:36.435 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.483 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:36 +0000 (0:00:00.062) 0:02:36.313 ********* 2020-07-23T12:24:36.494 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.538 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:36 +0000 (0:00:00.054) 0:02:36.368 ********* 2020-07-23T12:24:36.548 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-07-23T12:24:36.550 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.592 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:36 +0000 (0:00:00.054) 0:02:36.422 ********* 2020-07-23T12:24:36.604 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.644 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:36 +0000 (0:00:00.051) 0:02:36.473 ********* 2020-07-23T12:24:36.653 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-07-23T12:24:36.654 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.700 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:36 +0000 (0:00:00.056) 0:02:36.530 ********* 2020-07-23T12:24:36.718 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.752 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:36 +0000 (0:00:00.052) 0:02:36.582 ********* 2020-07-23T12:24:36.771 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.802 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:36 +0000 (0:00:00.050) 0:02:36.632 ********* 2020-07-23T12:24:36.813 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.853 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:36 +0000 (0:00:00.050) 0:02:36.683 ********* 2020-07-23T12:24:36.862 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-07-23T12:24:36.865 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.910 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:36 +0000 (0:00:00.056) 0:02:36.740 ********* 2020-07-23T12:24:36.920 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:36.983 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:36 +0000 (0:00:00.073) 0:02:36.813 ********* 2020-07-23T12:24:36.993 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-07-23T12:24:36.995 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:37.044 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:37 +0000 (0:00:00.060) 0:02:36.874 ********* 2020-07-23T12:24:37.055 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:37.105 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:37 +0000 (0:00:00.061) 0:02:36.935 ********* 2020-07-23T12:24:37.115 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-07-23T12:24:37.117 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:37.159 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:37 +0000 (0:00:00.053) 0:02:36.989 ********* 2020-07-23T12:24:37.178 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:37.215 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:37 +0000 (0:00:00.056) 0:02:37.045 ********* 2020-07-23T12:24:37.226 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-07-23T12:24:37.228 INFO:teuthology.task.ansible.out:skipping: [smithi133.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:37.268 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:37 +0000 (0:00:00.052) 0:02:37.098 ********* 2020-07-23T12:24:37.280 INFO:teuthology.task.ansible.out:skipping: [smithi137.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-07-23T12:24:37.334 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:37 +0000 (0:00:00.066) 0:02:37.164 ********* 2020-07-23T12:24:37.336 INFO:teuthology.task.ansible.out: 2020-07-23T12:24:37.479 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-07-23T12:24:37.481 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010044", "end": "2020-07-23 12:24:37.511292", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-07-23 12:24:37.501248", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:37.575 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011042", "end": "2020-07-23 12:24:37.527405", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-07-23 12:24:37.516363", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:37.630 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.004225", "end": "2020-07-23 12:24:37.704119", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-07-23 12:24:37.699894", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:37.722 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005942", "end": "2020-07-23 12:24:37.711273", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-07-23 12:24:37.705331", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:37.766 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005557", "end": "2020-07-23 12:24:37.828464", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-07-23 12:24:37.822907", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:37.841 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005343", "end": "2020-07-23 12:24:37.844695", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-07-23 12:24:37.839352", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:37.886 INFO:teuthology.task.ansible.out:ok: [smithi133.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.007261", "end": "2020-07-23 12:24:37.951075", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-07-23 12:24:37.943814", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:37.936 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:37 +0000 (0:00:00.602) 0:02:37.766 ********* 2020-07-23T12:24:37.968 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-07-23T12:24:38.006 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:38 +0000 (0:00:00.070) 0:02:37.836 ********* 2020-07-23T12:24:38.062 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-07-23T12:24:38.064 INFO:teuthology.task.ansible.out:ok: [smithi137.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.088609", "end": "2020-07-23 12:24:38.065226", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-07-23 12:24:37.976617", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-07-23T12:24:38.107 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:38 +0000 (0:00:00.100) 0:02:37.937 ********* 2020-07-23T12:24:38.150 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-07-23T12:24:38.194 INFO:teuthology.task.ansible.out:Thursday 23 July 2020 12:24:38 +0000 (0:00:00.087) 0:02:38.024 ********* 2020-07-23T12:24:38.205 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-07-23T12:24:38.206 INFO:teuthology.task.ansible.out:changed: [smithi133.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-07-23T12:24:38.345 INFO:teuthology.task.ansible.out:changed: [smithi137.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-07-23T12:24:38.446 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-07-23T12:24:38.450 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-07-23T12:24:38.454 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2020-07-23T12:24:38.456 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi133.front.sepia.ceph.com : ok=114 changed=31 unreachable=0 failed=0 skipped=175 rescued=0 ignored=0 smithi137.front.sepia.ceph.com : ok=114 changed=31 unreachable=0 failed=0 skipped=175 rescued=0 ignored=0 Thursday 23 July 2020 12:24:38 +0000 (0:00:00.260) 0:02:38.285 ********* =============================================================================== users : Create all admin users with sudo access. ----------------------- 21.66s users : Update authorized_keys using the keys repo --------------------- 19.81s testnode : Install packages --------------------------------------------- 8.57s testnode : Zap all non-root disks --------------------------------------- 8.56s testnode : Ensure packages are not present. ----------------------------- 8.48s users : Remove revoked users -------------------------------------------- 6.33s common : Update apt cache ----------------------------------------------- 3.66s testnode : Update apt cache. -------------------------------------------- 2.88s Gathering Facts --------------------------------------------------------- 2.00s testnode : Install apt keys --------------------------------------------- 1.96s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.65s testnode : Create logical volume(s) ------------------------------------- 1.58s users : Update authorized_keys for each user with literal keys ---------- 1.53s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.22s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.10s testnode : Ensure perl-doc is installed on apt systems. ----------------- 1.01s common : Set the hardware clock ----------------------------------------- 1.00s users : Clone the keys repo --------------------------------------------- 0.96s testnode : Inst 2020-07-23T12:24:38.457 INFO:teuthology.task.ansible.out:all non aarch64 packages --------------------------------- 0.95s common : Install nrpe package and dependencies (Ubuntu/Fedora) ---------- 0.95s 2020-07-23T12:24:38.865 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi137.front.sepia.ceph.com'), Remote(name='ubuntu@smithi133.front.sepia.ceph.com')] 2020-07-23T12:24:38.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-23T12:24:38.869 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:24:38.995 INFO:teuthology.orchestra.run.smithi137:> true 2020-07-23T12:24:39.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-07-23T12:24:39.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-23T12:24:39.344 INFO:teuthology.orchestra.run.smithi133:> true 2020-07-23T12:24:39.546 INFO:teuthology.run_tasks:Running task clock... 2020-07-23T12:24:39.563 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-07-23T12:24:39.563 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-07-23T12:24:39.564 INFO:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-07-23T12:24:39.615 INFO:teuthology.orchestra.run.smithi137.stdout:23 Jul 12:24:39 ntpd[7860]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-07-23T12:24:39.615 INFO:teuthology.orchestra.run.smithi137.stdout:23 Jul 12:24:39 ntpd[7860]: Command line: ntpd -gq 2020-07-23T12:24:39.617 INFO:teuthology.orchestra.run.smithi137.stdout:23 Jul 12:24:39 ntpd[7860]: proto: precision = 0.067 usec (-24) 2020-07-23T12:24:39.617 INFO:teuthology.orchestra.run.smithi137.stdout:23 Jul 12:24:39 ntpd[7860]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-07-23T12:24:39.618 INFO:teuthology.orchestra.run.smithi137.stdout:23 Jul 12:24:39 ntpd[7860]: restrict ::: KOD does nothing without LIMITED. 2020-07-23T12:24:39.618 INFO:teuthology.orchestra.run.smithi137.stdout:23 Jul 12:24:39 ntpd[7860]: Listen and drop on 0 v6wildcard [::]:123 2020-07-23T12:24:39.618 INFO:teuthology.orchestra.run.smithi137.stdout:23 Jul 12:24:39 ntpd[7860]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-07-23T12:24:39.618 INFO:teuthology.orchestra.run.smithi137.stdout:23 Jul 12:24:39 ntpd[7860]: Listen normally on 2 lo 127.0.0.1:123 2020-07-23T12:24:39.618 INFO:teuthology.orchestra.run.smithi137.stdout:23 Jul 12:24:39 ntpd[7860]: Listen normally on 3 enp3s0f1 172.21.15.137:123 2020-07-23T12:24:39.619 INFO:teuthology.orchestra.run.smithi137.stdout:23 Jul 12:24:39 ntpd[7860]: Listen normally on 4 lo [::1]:123 2020-07-23T12:24:39.619 INFO:teuthology.orchestra.run.smithi137.stdout:23 Jul 12:24:39 ntpd[7860]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8137%5]:123 2020-07-23T12:24:39.619 INFO:teuthology.orchestra.run.smithi137.stdout:23 Jul 12:24:39 ntpd[7860]: Listening on routing socket on fd #22 for interface updates 2020-07-23T12:24:39.620 INFO:teuthology.orchestra.run.smithi137.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-07-23T12:24:39.620 INFO:teuthology.orchestra.run.smithi137.stderr:restrict ::: KOD does nothing without LIMITED. 2020-07-23T12:24:40.619 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:40 ntpd[7860]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:40.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:40 ntpd[7860]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:40.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:40 ntpd[7860]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:40.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:40 ntpd[7860]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:40.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:40 ntpd[7860]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:40.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:40 ntpd[7860]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:40.621 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:40 ntpd[7860]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:40.621 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:40 ntpd[7860]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:42.618 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:42 ntpd[7860]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:42.618 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:42 ntpd[7860]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:42.618 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:42 ntpd[7860]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:42.619 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:42 ntpd[7860]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:42.619 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:42 ntpd[7860]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:42.619 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:42 ntpd[7860]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:42.619 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:42 ntpd[7860]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:42.619 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:42 ntpd[7860]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:42.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:42 ntpd[7860]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:42.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:42 ntpd[7860]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:42.621 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:42 ntpd[7860]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:42.621 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:42 ntpd[7860]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:43.619 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:43 ntpd[7860]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:43.619 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:43 ntpd[7860]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:43.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:43 ntpd[7860]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:43.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:43 ntpd[7860]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:44.619 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:44 ntpd[7860]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:44.619 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:44 ntpd[7860]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:44.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:44 ntpd[7860]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:44.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:44 ntpd[7860]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:44.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:44 ntpd[7860]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:44.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:44 ntpd[7860]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:44.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:44 ntpd[7860]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:44.621 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:44 ntpd[7860]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:44.621 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:44 ntpd[7860]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:44.621 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:44 ntpd[7860]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:44.621 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:44 ntpd[7860]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:44.621 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:44 ntpd[7860]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:45.618 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:45 ntpd[7860]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:45.619 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:45 ntpd[7860]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:45.619 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:45 ntpd[7860]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:45.619 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:45 ntpd[7860]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:46.619 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:46 ntpd[7860]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:46.619 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:46 ntpd[7860]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:46.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:46 ntpd[7860]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:46.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:46 ntpd[7860]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:46.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:46 ntpd[7860]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-07-23T12:24:46.620 INFO:teuthology.orchestra.run.smithi137.stderr:23 Jul 12:24:46 ntpd[7860]: can't open /var/log/ntpstats/loopstats.20200723: Permission denied 2020-07-23T12:24:46.620 INFO:teuthology.orchestra.run.smithi137.stdout:23 Jul 12:24:46 ntpd[7860]: ntpd: time set -0.028623 s 2020-07-23T12:24:46.621 INFO:teuthology.orchestra.run.smithi137.stdout:ntpd: time set -0.028623s 2020-07-23T12:24:46.739 INFO:teuthology.orchestra.run.smithi137.stdout: remote refid st t when poll reach delay offset jitter 2020-07-23T12:24:46.739 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================== 2020-07-23T12:24:46.739 INFO:teuthology.orchestra.run.smithi137.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-23T12:24:46.740 INFO:teuthology.orchestra.run.smithi137.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-23T12:24:46.740 INFO:teuthology.orchestra.run.smithi137.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-23T12:24:46.740 INFO:teuthology.orchestra.run.smithi137.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-23T12:24:46.741 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-07-23T12:24:46.741 INFO:teuthology.orchestra.run.smithi133:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-07-23T12:24:46.791 INFO:teuthology.orchestra.run.smithi133.stdout:23 Jul 12:24:46 ntpd[7776]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-07-23T12:24:46.791 INFO:teuthology.orchestra.run.smithi133.stdout:23 Jul 12:24:46 ntpd[7776]: Command line: ntpd -gq 2020-07-23T12:24:46.793 INFO:teuthology.orchestra.run.smithi133.stdout:23 Jul 12:24:46 ntpd[7776]: proto: precision = 0.041 usec (-24) 2020-07-23T12:24:46.793 INFO:teuthology.orchestra.run.smithi133.stdout:23 Jul 12:24:46 ntpd[7776]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-07-23T12:24:46.793 INFO:teuthology.orchestra.run.smithi133.stdout:23 Jul 12:24:46 ntpd[7776]: restrict ::: KOD does nothing without LIMITED. 2020-07-23T12:24:46.793 INFO:teuthology.orchestra.run.smithi133.stdout:23 Jul 12:24:46 ntpd[7776]: Listen and drop on 0 v6wildcard [::]:123 2020-07-23T12:24:46.794 INFO:teuthology.orchestra.run.smithi133.stdout:23 Jul 12:24:46 ntpd[7776]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-07-23T12:24:46.794 INFO:teuthology.orchestra.run.smithi133.stdout:23 Jul 12:24:46 ntpd[7776]: Listen normally on 2 lo 127.0.0.1:123 2020-07-23T12:24:46.794 INFO:teuthology.orchestra.run.smithi133.stdout:23 Jul 12:24:46 ntpd[7776]: Listen normally on 3 enp3s0f1 172.21.15.133:123 2020-07-23T12:24:46.794 INFO:teuthology.orchestra.run.smithi133.stdout:23 Jul 12:24:46 ntpd[7776]: Listen normally on 4 lo [::1]:123 2020-07-23T12:24:46.794 INFO:teuthology.orchestra.run.smithi133.stdout:23 Jul 12:24:46 ntpd[7776]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8087%5]:123 2020-07-23T12:24:46.795 INFO:teuthology.orchestra.run.smithi133.stdout:23 Jul 12:24:46 ntpd[7776]: Listening on routing socket on fd #22 for interface updates 2020-07-23T12:24:46.795 INFO:teuthology.orchestra.run.smithi133.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-07-23T12:24:46.795 INFO:teuthology.orchestra.run.smithi133.stderr:restrict ::: KOD does nothing without LIMITED. 2020-07-23T12:24:47.794 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:47 ntpd[7776]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:47.794 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:47 ntpd[7776]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:47.794 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:47 ntpd[7776]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:47.795 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:47 ntpd[7776]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:47.795 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:47 ntpd[7776]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:47.795 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:47 ntpd[7776]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:47.795 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:47 ntpd[7776]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:47.795 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:47 ntpd[7776]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:47.795 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:47 ntpd[7776]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:47.796 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:47 ntpd[7776]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:47.796 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:47 ntpd[7776]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:47.796 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:47 ntpd[7776]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:49.794 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:49 ntpd[7776]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:49.794 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:49 ntpd[7776]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:49.795 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:49 ntpd[7776]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:49.795 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:49 ntpd[7776]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:49.795 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:49 ntpd[7776]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:49.795 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:49 ntpd[7776]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:49.795 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:49 ntpd[7776]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:49.795 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:49 ntpd[7776]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:49.796 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:49 ntpd[7776]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:49.796 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:49 ntpd[7776]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:49.796 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:49 ntpd[7776]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:49.796 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:49 ntpd[7776]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:51.796 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:51 ntpd[7776]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:51.797 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:51 ntpd[7776]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:51.797 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:51 ntpd[7776]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:51.797 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:51 ntpd[7776]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:51.798 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:51 ntpd[7776]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:51.798 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:51 ntpd[7776]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:51.798 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:51 ntpd[7776]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:51.799 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:51 ntpd[7776]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:51.800 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:51 ntpd[7776]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:51.800 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:51 ntpd[7776]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:51.801 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:51 ntpd[7776]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:51.801 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:51 ntpd[7776]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:51.802 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:51 ntpd[7776]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:51.802 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:51 ntpd[7776]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:51.802 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:51 ntpd[7776]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:51.803 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:51 ntpd[7776]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:53.796 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:53 ntpd[7776]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:53.797 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:53 ntpd[7776]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:53.797 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:53 ntpd[7776]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:53.797 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:53 ntpd[7776]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:53.797 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:53 ntpd[7776]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-07-23T12:24:53.798 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:53 ntpd[7776]: can't open /var/log/ntpstats/rawstats.20200723: Permission denied 2020-07-23T12:24:53.798 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:53 ntpd[7776]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-07-23T12:24:53.798 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:53 ntpd[7776]: can't open /var/log/ntpstats/peerstats.20200723: Permission denied 2020-07-23T12:24:53.798 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:53 ntpd[7776]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-07-23T12:24:53.799 INFO:teuthology.orchestra.run.smithi133.stderr:23 Jul 12:24:53 ntpd[7776]: can't open /var/log/ntpstats/loopstats.20200723: Permission denied 2020-07-23T12:24:53.799 INFO:teuthology.orchestra.run.smithi133.stdout:23 Jul 12:24:53 ntpd[7776]: ntpd: time set -0.090572 s 2020-07-23T12:24:53.799 INFO:teuthology.orchestra.run.smithi133.stdout:ntpd: time set -0.090572s 2020-07-23T12:24:53.856 INFO:teuthology.orchestra.run.smithi133.stdout: remote refid st t when poll reach delay offset jitter 2020-07-23T12:24:53.857 INFO:teuthology.orchestra.run.smithi133.stdout:============================================================================== 2020-07-23T12:24:53.857 INFO:teuthology.orchestra.run.smithi133.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-23T12:24:53.858 INFO:teuthology.orchestra.run.smithi133.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-23T12:24:53.858 INFO:teuthology.orchestra.run.smithi133.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-23T12:24:53.861 INFO:teuthology.orchestra.run.smithi133.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-07-23T12:24:53.863 INFO:teuthology.run_tasks:Running task install... 2020-07-23T12:24:53.878 DEBUG:teuthology.task.install:project ceph 2020-07-23T12:24:53.879 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac'}} 2020-07-23T12:24:53.880 DEBUG:teuthology.task.install:config {'sha1': 'bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac'} 2020-07-23T12:24:53.880 INFO:teuthology.task.install:Using flavor: basic 2020-07-23T12:24:53.906 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-07-23T12:24:53.907 INFO:teuthology.task.install:extra packages: [] 2020-07-23T12:24:53.908 INFO:teuthology.orchestra.run.smithi137:> sudo lsb_release -is 2020-07-23T12:24:53.970 INFO:teuthology.orchestra.run.smithi137.stdout:Ubuntu 2020-07-23T12:24:53.974 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-23T12:24:53.976 INFO:teuthology.orchestra.run.smithi133:> sudo lsb_release -is 2020-07-23T12:24:53.977 INFO:teuthology.orchestra.run.smithi137:> sudo apt-key list | grep Ceph 2020-07-23T12:24:54.030 INFO:teuthology.orchestra.run.smithi133.stdout:Ubuntu 2020-07-23T12:24:54.037 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-23T12:24:54.038 INFO:teuthology.orchestra.run.smithi133:> sudo apt-key list | grep Ceph 2020-07-23T12:24:54.076 INFO:teuthology.orchestra.run.smithi137.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-07-23T12:24:54.121 INFO:teuthology.orchestra.run.smithi133.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-07-23T12:24:54.157 INFO:teuthology.orchestra.run.smithi137.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-07-23T12:24:54.161 INFO:teuthology.orchestra.run.smithi137.stdout:uid [ unknown] Ceph.com (release key) 2020-07-23T12:24:54.166 INFO:teuthology.orchestra.run.smithi137:> sudo lsb_release -is 2020-07-23T12:24:54.179 INFO:teuthology.orchestra.run.smithi133.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-07-23T12:24:54.180 INFO:teuthology.orchestra.run.smithi133.stdout:uid [ unknown] Ceph.com (release key) 2020-07-23T12:24:54.181 INFO:teuthology.orchestra.run.smithi133:> sudo lsb_release -is 2020-07-23T12:24:54.229 INFO:teuthology.orchestra.run.smithi137.stdout:Ubuntu 2020-07-23T12:24:54.234 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-23T12:24:54.234 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-07-23T12:24:54.235 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac 2020-07-23T12:24:54.241 INFO:teuthology.orchestra.run.smithi133.stdout:Ubuntu 2020-07-23T12:24:54.242 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-23T12:24:54.243 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-07-23T12:24:54.244 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac 2020-07-23T12:24:54.425 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default/ 2020-07-23T12:24:54.425 INFO:teuthology.task.install.deb:Package version is 15.2.4-377-gbdc0fb0-1bionic 2020-07-23T12:24:54.432 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default/ 2020-07-23T12:24:54.432 INFO:teuthology.task.install.deb:Package version is 15.2.4-377-gbdc0fb0-1bionic 2020-07-23T12:24:54.671 INFO:teuthology.orchestra.run.smithi137:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2020-07-23T12:24:54.720 INFO:teuthology.orchestra.run.smithi137:> sudo apt-get update 2020-07-23T12:24:54.789 INFO:teuthology.orchestra.run.smithi133:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2020-07-23T12:24:54.843 INFO:teuthology.orchestra.run.smithi133:> sudo apt-get update 2020-07-23T12:24:54.923 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-07-23T12:24:55.019 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-07-23T12:24:55.096 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-07-23T12:24:55.102 INFO:teuthology.orchestra.run.smithi137.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic InRelease 2020-07-23T12:24:55.153 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-07-23T12:24:55.153 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-07-23T12:24:55.154 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-07-23T12:24:55.184 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:6 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-07-23T12:24:55.201 INFO:teuthology.orchestra.run.smithi137.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic Release.gpg 2020-07-23T12:24:55.214 INFO:teuthology.orchestra.run.smithi133.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic InRelease 2020-07-23T12:24:55.218 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-07-23T12:24:55.253 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.8 kB] 2020-07-23T12:24:55.270 INFO:teuthology.orchestra.run.smithi133.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-07-23T12:24:55.318 INFO:teuthology.orchestra.run.smithi133.stdout:Hit:6 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-07-23T12:24:55.319 INFO:teuthology.orchestra.run.smithi133.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic Release.gpg 2020-07-23T12:24:55.319 INFO:teuthology.orchestra.run.smithi137.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,074 B] 2020-07-23T12:24:55.366 INFO:teuthology.orchestra.run.smithi133.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,074 B] 2020-07-23T12:24:55.448 INFO:teuthology.orchestra.run.smithi133.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.8 kB] 2020-07-23T12:24:55.493 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 26.5 kB in 1s (46.4 kB/s) 2020-07-23T12:24:55.753 INFO:teuthology.orchestra.run.smithi133.stdout:Fetched 26.5 kB in 1s (37.7 kB/s) 2020-07-23T12:24:56.693 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2020-07-23T12:24:56.741 INFO:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.4-377-gbdc0fb0-1bionic cephadm=15.2.4-377-gbdc0fb0-1bionic ceph-mds=15.2.4-377-gbdc0fb0-1bionic ceph-mgr=15.2.4-377-gbdc0fb0-1bionic ceph-common=15.2.4-377-gbdc0fb0-1bionic ceph-fuse=15.2.4-377-gbdc0fb0-1bionic ceph-test=15.2.4-377-gbdc0fb0-1bionic radosgw=15.2.4-377-gbdc0fb0-1bionic python3-rados=15.2.4-377-gbdc0fb0-1bionic python3-rgw=15.2.4-377-gbdc0fb0-1bionic python3-cephfs=15.2.4-377-gbdc0fb0-1bionic python3-rbd=15.2.4-377-gbdc0fb0-1bionic libcephfs2=15.2.4-377-gbdc0fb0-1bionic libcephfs-dev=15.2.4-377-gbdc0fb0-1bionic librados2=15.2.4-377-gbdc0fb0-1bionic librbd1=15.2.4-377-gbdc0fb0-1bionic rbd-fuse=15.2.4-377-gbdc0fb0-1bionic 2020-07-23T12:24:56.825 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2020-07-23T12:24:56.945 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2020-07-23T12:24:56.990 INFO:teuthology.orchestra.run.smithi133:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.4-377-gbdc0fb0-1bionic cephadm=15.2.4-377-gbdc0fb0-1bionic ceph-mds=15.2.4-377-gbdc0fb0-1bionic ceph-mgr=15.2.4-377-gbdc0fb0-1bionic ceph-common=15.2.4-377-gbdc0fb0-1bionic ceph-fuse=15.2.4-377-gbdc0fb0-1bionic ceph-test=15.2.4-377-gbdc0fb0-1bionic radosgw=15.2.4-377-gbdc0fb0-1bionic python3-rados=15.2.4-377-gbdc0fb0-1bionic python3-rgw=15.2.4-377-gbdc0fb0-1bionic python3-cephfs=15.2.4-377-gbdc0fb0-1bionic python3-rbd=15.2.4-377-gbdc0fb0-1bionic libcephfs2=15.2.4-377-gbdc0fb0-1bionic libcephfs-dev=15.2.4-377-gbdc0fb0-1bionic librados2=15.2.4-377-gbdc0fb0-1bionic librbd1=15.2.4-377-gbdc0fb0-1bionic rbd-fuse=15.2.4-377-gbdc0fb0-1bionic 2020-07-23T12:24:57.066 INFO:teuthology.orchestra.run.smithi133.stdout:Reading package lists... 2020-07-23T12:24:57.070 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2020-07-23T12:24:57.070 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2020-07-23T12:24:57.231 INFO:teuthology.orchestra.run.smithi137.stdout:The following additional packages will be installed: 2020-07-23T12:24:57.232 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-07-23T12:24:57.232 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-07-23T12:24:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2020-07-23T12:24:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2020-07-23T12:24:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: libsnappy1v5 libxslt1.1 python-attr python-funcsigs python-pastedeploy-tpl 2020-07-23T12:24:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-07-23T12:24:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-07-23T12:24:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-07-23T12:24:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-07-23T12:24:57.237 INFO:teuthology.orchestra.run.smithi137.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2020-07-23T12:24:57.238 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2020-07-23T12:24:57.238 INFO:teuthology.orchestra.run.smithi137.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2020-07-23T12:24:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2020-07-23T12:24:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2020-07-23T12:24:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-07-23T12:24:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-07-23T12:24:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytest python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-07-23T12:24:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-07-23T12:24:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-07-23T12:24:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-07-23T12:24:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-07-23T12:24:57.243 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-07-23T12:24:57.244 INFO:teuthology.orchestra.run.smithi137.stdout:Suggested packages: 2020-07-23T12:24:57.244 INFO:teuthology.orchestra.run.smithi137.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-07-23T12:24:57.244 INFO:teuthology.orchestra.run.smithi137.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-07-23T12:24:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-07-23T12:24:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-07-23T12:24:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-07-23T12:24:57.246 INFO:teuthology.orchestra.run.smithi137.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-07-23T12:24:57.246 INFO:teuthology.orchestra.run.smithi137.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-07-23T12:24:57.246 INFO:teuthology.orchestra.run.smithi137.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-07-23T12:24:57.247 INFO:teuthology.orchestra.run.smithi137.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-07-23T12:24:57.247 INFO:teuthology.orchestra.run.smithi137.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-07-23T12:24:57.247 INFO:teuthology.orchestra.run.smithi137.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-07-23T12:24:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-07-23T12:24:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-07-23T12:24:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-07-23T12:24:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-07-23T12:24:57.321 INFO:teuthology.orchestra.run.smithi133.stdout:Building dependency tree... 2020-07-23T12:24:57.322 INFO:teuthology.orchestra.run.smithi133.stdout:Reading state information... 2020-07-23T12:24:57.337 INFO:teuthology.orchestra.run.smithi137.stdout:The following NEW packages will be installed: 2020-07-23T12:24:57.337 INFO:teuthology.orchestra.run.smithi137.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-07-23T12:24:57.337 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-07-23T12:24:57.337 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-07-23T12:24:57.338 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2020-07-23T12:24:57.338 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-07-23T12:24:57.338 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-07-23T12:24:57.339 INFO:teuthology.orchestra.run.smithi137.stdout: python-funcsigs python-pastedeploy-tpl python-pluggy python-py python-pytest 2020-07-23T12:24:57.339 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-07-23T12:24:57.339 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2020-07-23T12:24:57.339 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-07-23T12:24:57.340 INFO:teuthology.orchestra.run.smithi137.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-07-23T12:24:57.340 INFO:teuthology.orchestra.run.smithi137.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2020-07-23T12:24:57.340 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2020-07-23T12:24:57.340 INFO:teuthology.orchestra.run.smithi137.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2020-07-23T12:24:57.340 INFO:teuthology.orchestra.run.smithi137.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2020-07-23T12:24:57.341 INFO:teuthology.orchestra.run.smithi137.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2020-07-23T12:24:57.341 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-07-23T12:24:57.341 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-07-23T12:24:57.341 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2020-07-23T12:24:57.341 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2020-07-23T12:24:57.341 INFO:teuthology.orchestra.run.smithi137.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2020-07-23T12:24:57.343 INFO:teuthology.orchestra.run.smithi137.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2020-07-23T12:24:57.343 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2020-07-23T12:24:57.344 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2020-07-23T12:24:57.344 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-07-23T12:24:57.344 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages will be upgraded: 2020-07-23T12:24:57.344 INFO:teuthology.orchestra.run.smithi137.stdout: librados2 librbd1 2020-07-23T12:24:57.488 INFO:teuthology.orchestra.run.smithi133.stdout:The following additional packages will be installed: 2020-07-23T12:24:57.488 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-07-23T12:24:57.489 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-07-23T12:24:57.490 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2020-07-23T12:24:57.490 INFO:teuthology.orchestra.run.smithi133.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2020-07-23T12:24:57.490 INFO:teuthology.orchestra.run.smithi133.stdout: libsnappy1v5 libxslt1.1 python-attr python-funcsigs python-pastedeploy-tpl 2020-07-23T12:24:57.490 INFO:teuthology.orchestra.run.smithi133.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-07-23T12:24:57.491 INFO:teuthology.orchestra.run.smithi133.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-07-23T12:24:57.491 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-07-23T12:24:57.491 INFO:teuthology.orchestra.run.smithi133.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-07-23T12:24:57.491 INFO:teuthology.orchestra.run.smithi133.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2020-07-23T12:24:57.491 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2020-07-23T12:24:57.491 INFO:teuthology.orchestra.run.smithi133.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2020-07-23T12:24:57.492 INFO:teuthology.orchestra.run.smithi133.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2020-07-23T12:24:57.492 INFO:teuthology.orchestra.run.smithi133.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2020-07-23T12:24:57.492 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-07-23T12:24:57.493 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-07-23T12:24:57.493 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytest python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-07-23T12:24:57.494 INFO:teuthology.orchestra.run.smithi133.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-07-23T12:24:57.494 INFO:teuthology.orchestra.run.smithi133.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-07-23T12:24:57.494 INFO:teuthology.orchestra.run.smithi133.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-07-23T12:24:57.495 INFO:teuthology.orchestra.run.smithi133.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-07-23T12:24:57.496 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-07-23T12:24:57.497 INFO:teuthology.orchestra.run.smithi133.stdout:Suggested packages: 2020-07-23T12:24:57.498 INFO:teuthology.orchestra.run.smithi133.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-07-23T12:24:57.498 INFO:teuthology.orchestra.run.smithi133.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-07-23T12:24:57.498 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-07-23T12:24:57.498 INFO:teuthology.orchestra.run.smithi133.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-07-23T12:24:57.498 INFO:teuthology.orchestra.run.smithi133.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-07-23T12:24:57.499 INFO:teuthology.orchestra.run.smithi133.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-07-23T12:24:57.499 INFO:teuthology.orchestra.run.smithi133.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-07-23T12:24:57.499 INFO:teuthology.orchestra.run.smithi133.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-07-23T12:24:57.499 INFO:teuthology.orchestra.run.smithi133.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-07-23T12:24:57.499 INFO:teuthology.orchestra.run.smithi133.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-07-23T12:24:57.500 INFO:teuthology.orchestra.run.smithi133.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-07-23T12:24:57.500 INFO:teuthology.orchestra.run.smithi133.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-07-23T12:24:57.500 INFO:teuthology.orchestra.run.smithi133.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-07-23T12:24:57.500 INFO:teuthology.orchestra.run.smithi133.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-07-23T12:24:57.500 INFO:teuthology.orchestra.run.smithi133.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-07-23T12:24:57.528 INFO:teuthology.orchestra.run.smithi137.stdout:2 upgraded, 110 newly installed, 0 to remove and 14 not upgraded. 2020-07-23T12:24:57.529 INFO:teuthology.orchestra.run.smithi137.stdout:Need to get 157 MB of archives. 2020-07-23T12:24:57.529 INFO:teuthology.orchestra.run.smithi137.stdout:After this operation, 729 MB of additional disk space will be used. 2020-07-23T12:24:57.529 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-07-23T12:24:57.595 INFO:teuthology.orchestra.run.smithi133.stdout:The following NEW packages will be installed: 2020-07-23T12:24:57.595 INFO:teuthology.orchestra.run.smithi133.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-07-23T12:24:57.596 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-07-23T12:24:57.596 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-07-23T12:24:57.596 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2020-07-23T12:24:57.598 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-07-23T12:24:57.598 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-07-23T12:24:57.599 INFO:teuthology.orchestra.run.smithi133.stdout: python-funcsigs python-pastedeploy-tpl python-pluggy python-py python-pytest 2020-07-23T12:24:57.599 INFO:teuthology.orchestra.run.smithi133.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-07-23T12:24:57.599 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2020-07-23T12:24:57.600 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-07-23T12:24:57.600 INFO:teuthology.orchestra.run.smithi133.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-07-23T12:24:57.600 INFO:teuthology.orchestra.run.smithi133.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2020-07-23T12:24:57.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2020-07-23T12:24:57.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2020-07-23T12:24:57.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2020-07-23T12:24:57.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2020-07-23T12:24:57.601 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-07-23T12:24:57.602 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-07-23T12:24:57.602 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2020-07-23T12:24:57.602 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2020-07-23T12:24:57.602 INFO:teuthology.orchestra.run.smithi133.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2020-07-23T12:24:57.602 INFO:teuthology.orchestra.run.smithi133.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2020-07-23T12:24:57.603 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2020-07-23T12:24:57.603 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2020-07-23T12:24:57.603 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-07-23T12:24:57.603 INFO:teuthology.orchestra.run.smithi133.stdout:The following packages will be upgraded: 2020-07-23T12:24:57.603 INFO:teuthology.orchestra.run.smithi133.stdout: librados2 librbd1 2020-07-23T12:24:57.604 INFO:teuthology.orchestra.run.smithi137.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.2.4-377-gbdc0fb0-1bionic [1,618 kB] 2020-07-23T12:24:57.609 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-07-23T12:24:57.787 INFO:teuthology.orchestra.run.smithi133.stdout:2 upgraded, 110 newly installed, 0 to remove and 14 not upgraded. 2020-07-23T12:24:57.787 INFO:teuthology.orchestra.run.smithi133.stdout:Need to get 157 MB of archives. 2020-07-23T12:24:57.787 INFO:teuthology.orchestra.run.smithi133.stdout:After this operation, 729 MB of additional disk space will be used. 2020-07-23T12:24:57.788 INFO:teuthology.orchestra.run.smithi133.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-07-23T12:24:57.858 INFO:teuthology.orchestra.run.smithi133.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.2.4-377-gbdc0fb0-1bionic [1,618 kB] 2020-07-23T12:24:57.866 INFO:teuthology.orchestra.run.smithi133.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-07-23T12:24:57.876 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-07-23T12:24:57.876 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-07-23T12:24:57.889 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-07-23T12:24:57.984 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-07-23T12:24:57.986 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-07-23T12:24:58.026 INFO:teuthology.orchestra.run.smithi137.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-07-23T12:24:58.034 INFO:teuthology.orchestra.run.smithi137.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-07-23T12:24:58.042 INFO:teuthology.orchestra.run.smithi137.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-07-23T12:24:58.051 INFO:teuthology.orchestra.run.smithi137.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-07-23T12:24:58.061 INFO:teuthology.orchestra.run.smithi137.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] 2020-07-23T12:24:58.082 INFO:teuthology.orchestra.run.smithi137.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-07-23T12:24:58.090 INFO:teuthology.orchestra.run.smithi137.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-07-23T12:24:58.113 INFO:teuthology.orchestra.run.smithi137.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-07-23T12:24:58.114 INFO:teuthology.orchestra.run.smithi133.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-07-23T12:24:58.122 INFO:teuthology.orchestra.run.smithi133.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-07-23T12:24:58.124 INFO:teuthology.orchestra.run.smithi137.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-07-23T12:24:58.127 INFO:teuthology.orchestra.run.smithi137.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-07-23T12:24:58.128 INFO:teuthology.orchestra.run.smithi133.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-07-23T12:24:58.194 INFO:teuthology.orchestra.run.smithi137.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-07-23T12:24:58.194 INFO:teuthology.orchestra.run.smithi137.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-07-23T12:24:58.196 INFO:teuthology.orchestra.run.smithi137.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-07-23T12:24:58.197 INFO:teuthology.orchestra.run.smithi137.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-07-23T12:24:58.197 INFO:teuthology.orchestra.run.smithi137.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-07-23T12:24:58.201 INFO:teuthology.orchestra.run.smithi137.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-07-23T12:24:58.203 INFO:teuthology.orchestra.run.smithi137.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-07-23T12:24:58.206 INFO:teuthology.orchestra.run.smithi137.stdout:Get:26 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.2.4-377-gbdc0fb0-1bionic [3,049 kB] 2020-07-23T12:24:58.221 INFO:teuthology.orchestra.run.smithi133.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.2.4-377-gbdc0fb0-1bionic [3,049 kB] 2020-07-23T12:24:58.236 INFO:teuthology.orchestra.run.smithi133.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-07-23T12:24:58.239 INFO:teuthology.orchestra.run.smithi133.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-07-23T12:24:58.275 INFO:teuthology.orchestra.run.smithi137.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-07-23T12:24:58.276 INFO:teuthology.orchestra.run.smithi137.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-07-23T12:24:58.282 INFO:teuthology.orchestra.run.smithi133.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-07-23T12:24:58.288 INFO:teuthology.orchestra.run.smithi137.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-07-23T12:24:58.289 INFO:teuthology.orchestra.run.smithi133.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-07-23T12:24:58.302 INFO:teuthology.orchestra.run.smithi133.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-07-23T12:24:58.307 INFO:teuthology.orchestra.run.smithi133.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-07-23T12:24:58.315 INFO:teuthology.orchestra.run.smithi133.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] 2020-07-23T12:24:58.338 INFO:teuthology.orchestra.run.smithi133.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-07-23T12:24:58.342 INFO:teuthology.orchestra.run.smithi133.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-07-23T12:24:58.354 INFO:teuthology.orchestra.run.smithi137.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-07-23T12:24:58.355 INFO:teuthology.orchestra.run.smithi133.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-07-23T12:24:58.355 INFO:teuthology.orchestra.run.smithi137.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-07-23T12:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-07-23T12:24:58.360 INFO:teuthology.orchestra.run.smithi137.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-07-23T12:24:58.365 INFO:teuthology.orchestra.run.smithi137.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-07-23T12:24:58.366 INFO:teuthology.orchestra.run.smithi133.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-07-23T12:24:58.367 INFO:teuthology.orchestra.run.smithi137.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-07-23T12:24:58.368 INFO:teuthology.orchestra.run.smithi137.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-07-23T12:24:58.438 INFO:teuthology.orchestra.run.smithi137.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-07-23T12:24:58.439 INFO:teuthology.orchestra.run.smithi133.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-07-23T12:24:58.439 INFO:teuthology.orchestra.run.smithi133.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-07-23T12:24:58.439 INFO:teuthology.orchestra.run.smithi133.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-07-23T12:24:58.439 INFO:teuthology.orchestra.run.smithi133.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-07-23T12:24:58.440 INFO:teuthology.orchestra.run.smithi133.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-07-23T12:24:58.440 INFO:teuthology.orchestra.run.smithi133.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-07-23T12:24:58.441 INFO:teuthology.orchestra.run.smithi133.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-07-23T12:24:58.444 INFO:teuthology.orchestra.run.smithi133.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-07-23T12:24:58.513 INFO:teuthology.orchestra.run.smithi133.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-07-23T12:24:58.514 INFO:teuthology.orchestra.run.smithi133.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-07-23T12:24:58.517 INFO:teuthology.orchestra.run.smithi133.stdout:Get:29 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.2.4-377-gbdc0fb0-1bionic [433 kB] 2020-07-23T12:24:58.526 INFO:teuthology.orchestra.run.smithi137.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-07-23T12:24:58.527 INFO:teuthology.orchestra.run.smithi137.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-07-23T12:24:58.537 INFO:teuthology.orchestra.run.smithi133.stdout:Get:30 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.2.4-377-gbdc0fb0-1bionic [331 kB] 2020-07-23T12:24:58.585 INFO:teuthology.orchestra.run.smithi133.stdout:Get:31 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.2.4-377-gbdc0fb0-1bionic [20.6 kB] 2020-07-23T12:24:58.586 INFO:teuthology.orchestra.run.smithi133.stdout:Get:32 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.2.4-377-gbdc0fb0-1bionic [132 kB] 2020-07-23T12:24:58.592 INFO:teuthology.orchestra.run.smithi133.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-07-23T12:24:58.592 INFO:teuthology.orchestra.run.smithi133.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-07-23T12:24:58.593 INFO:teuthology.orchestra.run.smithi133.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-07-23T12:24:58.594 INFO:teuthology.orchestra.run.smithi133.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-07-23T12:24:58.595 INFO:teuthology.orchestra.run.smithi133.stdout:Get:37 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 15.2.4-377-gbdc0fb0-1bionic [25.7 kB] 2020-07-23T12:24:58.597 INFO:teuthology.orchestra.run.smithi133.stdout:Get:38 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 15.2.4-377-gbdc0fb0-1bionic [308 kB] 2020-07-23T12:24:58.597 INFO:teuthology.orchestra.run.smithi133.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-07-23T12:24:58.604 INFO:teuthology.orchestra.run.smithi133.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-07-23T12:24:58.608 INFO:teuthology.orchestra.run.smithi133.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-07-23T12:24:58.609 INFO:teuthology.orchestra.run.smithi133.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-07-23T12:24:58.649 INFO:teuthology.orchestra.run.smithi133.stdout:Get:43 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.2.4-377-gbdc0fb0-1bionic [5,359 kB] 2020-07-23T12:24:58.672 INFO:teuthology.orchestra.run.smithi133.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-07-23T12:24:58.751 INFO:teuthology.orchestra.run.smithi133.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-07-23T12:24:58.752 INFO:teuthology.orchestra.run.smithi133.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-07-23T12:24:58.799 INFO:teuthology.orchestra.run.smithi137.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-07-23T12:24:58.863 INFO:teuthology.orchestra.run.smithi137.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-07-23T12:24:58.866 INFO:teuthology.orchestra.run.smithi137.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-07-23T12:24:58.879 INFO:teuthology.orchestra.run.smithi137.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-07-23T12:24:58.880 INFO:teuthology.orchestra.run.smithi137.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-07-23T12:24:58.880 INFO:teuthology.orchestra.run.smithi137.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-07-23T12:24:58.881 INFO:teuthology.orchestra.run.smithi137.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-07-23T12:24:58.881 INFO:teuthology.orchestra.run.smithi137.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-07-23T12:24:58.882 INFO:teuthology.orchestra.run.smithi137.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-07-23T12:24:58.882 INFO:teuthology.orchestra.run.smithi137.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-07-23T12:24:58.961 INFO:teuthology.orchestra.run.smithi137.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-07-23T12:24:58.962 INFO:teuthology.orchestra.run.smithi137.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-07-23T12:24:58.977 INFO:teuthology.orchestra.run.smithi137.stdout:Get:52 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.2.4-377-gbdc0fb0-1bionic [433 kB] 2020-07-23T12:24:59.032 INFO:teuthology.orchestra.run.smithi133.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-07-23T12:24:59.041 INFO:teuthology.orchestra.run.smithi137.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-07-23T12:24:59.042 INFO:teuthology.orchestra.run.smithi137.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-07-23T12:24:59.047 INFO:teuthology.orchestra.run.smithi137.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-07-23T12:24:59.047 INFO:teuthology.orchestra.run.smithi137.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-07-23T12:24:59.048 INFO:teuthology.orchestra.run.smithi137.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-07-23T12:24:59.051 INFO:teuthology.orchestra.run.smithi137.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-07-23T12:24:59.054 INFO:teuthology.orchestra.run.smithi137.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-07-23T12:24:59.055 INFO:teuthology.orchestra.run.smithi137.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-07-23T12:24:59.055 INFO:teuthology.orchestra.run.smithi137.stdout:Get:61 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.2.4-377-gbdc0fb0-1bionic [331 kB] 2020-07-23T12:24:59.082 INFO:teuthology.orchestra.run.smithi133.stdout:Get:48 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 15.2.4-377-gbdc0fb0-1bionic [99.2 kB] 2020-07-23T12:24:59.083 INFO:teuthology.orchestra.run.smithi133.stdout:Get:49 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.2.4-377-gbdc0fb0-1bionic [334 kB] 2020-07-23T12:24:59.083 INFO:teuthology.orchestra.run.smithi133.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-07-23T12:24:59.085 INFO:teuthology.orchestra.run.smithi133.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-07-23T12:24:59.096 INFO:teuthology.orchestra.run.smithi133.stdout:Get:52 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.2.4-377-gbdc0fb0-1bionic [18.7 MB] 2020-07-23T12:24:59.111 INFO:teuthology.orchestra.run.smithi133.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-07-23T12:24:59.112 INFO:teuthology.orchestra.run.smithi133.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-07-23T12:24:59.112 INFO:teuthology.orchestra.run.smithi133.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-07-23T12:24:59.113 INFO:teuthology.orchestra.run.smithi133.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-07-23T12:24:59.114 INFO:teuthology.orchestra.run.smithi133.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-07-23T12:24:59.114 INFO:teuthology.orchestra.run.smithi133.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-07-23T12:24:59.115 INFO:teuthology.orchestra.run.smithi133.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-07-23T12:24:59.121 INFO:teuthology.orchestra.run.smithi137.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-07-23T12:24:59.125 INFO:teuthology.orchestra.run.smithi137.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-07-23T12:24:59.128 INFO:teuthology.orchestra.run.smithi137.stdout:Get:64 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.2.4-377-gbdc0fb0-1bionic [20.6 kB] 2020-07-23T12:24:59.129 INFO:teuthology.orchestra.run.smithi137.stdout:Get:65 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.2.4-377-gbdc0fb0-1bionic [132 kB] 2020-07-23T12:24:59.134 INFO:teuthology.orchestra.run.smithi137.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 15.2.4-377-gbdc0fb0-1bionic [25.7 kB] 2020-07-23T12:24:59.140 INFO:teuthology.orchestra.run.smithi137.stdout:Get:67 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 15.2.4-377-gbdc0fb0-1bionic [308 kB] 2020-07-23T12:24:59.191 INFO:teuthology.orchestra.run.smithi133.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-07-23T12:24:59.193 INFO:teuthology.orchestra.run.smithi133.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-07-23T12:24:59.200 INFO:teuthology.orchestra.run.smithi133.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-07-23T12:24:59.201 INFO:teuthology.orchestra.run.smithi133.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-07-23T12:24:59.202 INFO:teuthology.orchestra.run.smithi137.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-07-23T12:24:59.204 INFO:teuthology.orchestra.run.smithi137.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-07-23T12:24:59.204 INFO:teuthology.orchestra.run.smithi137.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-07-23T12:24:59.204 INFO:teuthology.orchestra.run.smithi137.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-07-23T12:24:59.206 INFO:teuthology.orchestra.run.smithi137.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-07-23T12:24:59.210 INFO:teuthology.orchestra.run.smithi137.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-07-23T12:24:59.211 INFO:teuthology.orchestra.run.smithi137.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-07-23T12:24:59.212 INFO:teuthology.orchestra.run.smithi137.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-07-23T12:24:59.270 INFO:teuthology.orchestra.run.smithi133.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-07-23T12:24:59.271 INFO:teuthology.orchestra.run.smithi133.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-07-23T12:24:59.271 INFO:teuthology.orchestra.run.smithi133.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-07-23T12:24:59.275 INFO:teuthology.orchestra.run.smithi133.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-07-23T12:24:59.278 INFO:teuthology.orchestra.run.smithi133.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-07-23T12:24:59.280 INFO:teuthology.orchestra.run.smithi133.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-07-23T12:24:59.282 INFO:teuthology.orchestra.run.smithi137.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-07-23T12:24:59.285 INFO:teuthology.orchestra.run.smithi137.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-07-23T12:24:59.349 INFO:teuthology.orchestra.run.smithi133.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-07-23T12:24:59.350 INFO:teuthology.orchestra.run.smithi137.stdout:Get:78 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.2.4-377-gbdc0fb0-1bionic [5,359 kB] 2020-07-23T12:24:59.354 INFO:teuthology.orchestra.run.smithi133.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-07-23T12:24:59.361 INFO:teuthology.orchestra.run.smithi133.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-07-23T12:24:59.362 INFO:teuthology.orchestra.run.smithi133.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-07-23T12:24:59.362 INFO:teuthology.orchestra.run.smithi137.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-07-23T12:24:59.364 INFO:teuthology.orchestra.run.smithi137.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-07-23T12:24:59.365 INFO:teuthology.orchestra.run.smithi137.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-07-23T12:24:59.428 INFO:teuthology.orchestra.run.smithi133.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-07-23T12:24:59.429 INFO:teuthology.orchestra.run.smithi133.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-07-23T12:24:59.430 INFO:teuthology.orchestra.run.smithi133.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-07-23T12:24:59.433 INFO:teuthology.orchestra.run.smithi133.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-07-23T12:24:59.434 INFO:teuthology.orchestra.run.smithi133.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-07-23T12:24:59.434 INFO:teuthology.orchestra.run.smithi133.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-07-23T12:24:59.465 INFO:teuthology.orchestra.run.smithi137.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-07-23T12:24:59.466 INFO:teuthology.orchestra.run.smithi137.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-07-23T12:24:59.466 INFO:teuthology.orchestra.run.smithi137.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-07-23T12:24:59.469 INFO:teuthology.orchestra.run.smithi137.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-07-23T12:24:59.469 INFO:teuthology.orchestra.run.smithi137.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-07-23T12:24:59.470 INFO:teuthology.orchestra.run.smithi137.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-07-23T12:24:59.471 INFO:teuthology.orchestra.run.smithi137.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-07-23T12:24:59.507 INFO:teuthology.orchestra.run.smithi133.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-07-23T12:24:59.510 INFO:teuthology.orchestra.run.smithi133.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-07-23T12:24:59.525 INFO:teuthology.orchestra.run.smithi137.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-07-23T12:24:59.526 INFO:teuthology.orchestra.run.smithi137.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-07-23T12:24:59.526 INFO:teuthology.orchestra.run.smithi133.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-07-23T12:24:59.527 INFO:teuthology.orchestra.run.smithi133.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-07-23T12:24:59.586 INFO:teuthology.orchestra.run.smithi133.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-07-23T12:24:59.687 INFO:teuthology.orchestra.run.smithi133.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-07-23T12:24:59.687 INFO:teuthology.orchestra.run.smithi133.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-07-23T12:24:59.688 INFO:teuthology.orchestra.run.smithi133.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-07-23T12:24:59.691 INFO:teuthology.orchestra.run.smithi133.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-07-23T12:24:59.691 INFO:teuthology.orchestra.run.smithi133.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-07-23T12:24:59.692 INFO:teuthology.orchestra.run.smithi133.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-07-23T12:24:59.692 INFO:teuthology.orchestra.run.smithi133.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-07-23T12:24:59.693 INFO:teuthology.orchestra.run.smithi133.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-07-23T12:24:59.694 INFO:teuthology.orchestra.run.smithi133.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-07-23T12:25:00.431 INFO:teuthology.orchestra.run.smithi137.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 15.2.4-377-gbdc0fb0-1bionic [99.2 kB] 2020-07-23T12:25:00.435 INFO:teuthology.orchestra.run.smithi137.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.2.4-377-gbdc0fb0-1bionic [334 kB] 2020-07-23T12:25:00.501 INFO:teuthology.orchestra.run.smithi137.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.2.4-377-gbdc0fb0-1bionic [18.7 MB] 2020-07-23T12:25:00.742 INFO:teuthology.orchestra.run.smithi133.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.2.4-377-gbdc0fb0-1bionic [5,144 kB] 2020-07-23T12:25:01.191 INFO:teuthology.orchestra.run.smithi133.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 15.2.4-377-gbdc0fb0-1bionic [147 kB] 2020-07-23T12:25:01.199 INFO:teuthology.orchestra.run.smithi133.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.2.4-377-gbdc0fb0-1bionic [1,279 kB] 2020-07-23T12:25:01.290 INFO:teuthology.orchestra.run.smithi133.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.2.4-377-gbdc0fb0-1bionic [5,942 kB] 2020-07-23T12:25:01.778 INFO:teuthology.orchestra.run.smithi133.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.2.4-377-gbdc0fb0-1bionic [22.7 MB] 2020-07-23T12:25:03.527 INFO:teuthology.orchestra.run.smithi133.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.2.4-377-gbdc0fb0-1bionic [3,868 B] 2020-07-23T12:25:03.527 INFO:teuthology.orchestra.run.smithi133.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.2.4-377-gbdc0fb0-1bionic [514 kB] 2020-07-23T12:25:03.575 INFO:teuthology.orchestra.run.smithi133.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.2.4-377-gbdc0fb0-1bionic [1,847 kB] 2020-07-23T12:25:03.681 INFO:teuthology.orchestra.run.smithi133.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 15.2.4-377-gbdc0fb0-1bionic [43.1 kB] 2020-07-23T12:25:03.712 INFO:teuthology.orchestra.run.smithi133.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 15.2.4-377-gbdc0fb0-1bionic [45.4 kB] 2020-07-23T12:25:03.714 INFO:teuthology.orchestra.run.smithi133.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.2.4-377-gbdc0fb0-1bionic [3,157 kB] 2020-07-23T12:25:03.948 INFO:teuthology.orchestra.run.smithi133.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.2.4-377-gbdc0fb0-1bionic [34.2 kB] 2020-07-23T12:25:03.949 INFO:teuthology.orchestra.run.smithi133.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.2.4-377-gbdc0fb0-1bionic [7,886 kB] 2020-07-23T12:25:04.486 INFO:teuthology.orchestra.run.smithi133.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.2.4-377-gbdc0fb0-1bionic [3,980 B] 2020-07-23T12:25:04.486 INFO:teuthology.orchestra.run.smithi133.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.2.4-377-gbdc0fb0-1bionic [22.4 kB] 2020-07-23T12:25:04.487 INFO:teuthology.orchestra.run.smithi133.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.2.4-377-gbdc0fb0-1bionic [43.4 MB] 2020-07-23T12:25:04.500 INFO:teuthology.orchestra.run.smithi137.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.2.4-377-gbdc0fb0-1bionic [5,144 kB] 2020-07-23T12:25:05.376 INFO:teuthology.orchestra.run.smithi137.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 15.2.4-377-gbdc0fb0-1bionic [147 kB] 2020-07-23T12:25:05.412 INFO:teuthology.orchestra.run.smithi137.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.2.4-377-gbdc0fb0-1bionic [1,279 kB] 2020-07-23T12:25:05.591 INFO:teuthology.orchestra.run.smithi137.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.2.4-377-gbdc0fb0-1bionic [5,942 kB] 2020-07-23T12:25:06.367 INFO:teuthology.orchestra.run.smithi137.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.2.4-377-gbdc0fb0-1bionic [22.7 MB] 2020-07-23T12:25:06.882 INFO:teuthology.orchestra.run.smithi133.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.2.4-377-gbdc0fb0-1bionic [17.9 kB] 2020-07-23T12:25:06.883 INFO:teuthology.orchestra.run.smithi133.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.2.4-377-gbdc0fb0-1bionic [8,512 kB] 2020-07-23T12:25:07.244 INFO:teuthology.orchestra.run.smithi133.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.2.4-377-gbdc0fb0-1bionic [62.6 kB] 2020-07-23T12:25:07.947 INFO:teuthology.orchestra.run.smithi133.stdout:Fetched 157 MB in 10s (16.3 MB/s) 2020-07-23T12:25:08.071 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-uritemplate. 2020-07-23T12:25:08.125 INFO:teuthology.orchestra.run.smithi133.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137579 files and directories currently installed.) 2020-07-23T12:25:08.126 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-07-23T12:25:08.162 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-07-23T12:25:08.498 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-07-23T12:25:08.512 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-07-23T12:25:08.598 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-07-23T12:25:08.631 INFO:teuthology.orchestra.run.smithi137.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.2.4-377-gbdc0fb0-1bionic [3,868 B] 2020-07-23T12:25:08.632 INFO:teuthology.orchestra.run.smithi137.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.2.4-377-gbdc0fb0-1bionic [514 kB] 2020-07-23T12:25:08.674 INFO:teuthology.orchestra.run.smithi137.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.2.4-377-gbdc0fb0-1bionic [1,847 kB] 2020-07-23T12:25:08.879 INFO:teuthology.orchestra.run.smithi137.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 15.2.4-377-gbdc0fb0-1bionic [43.1 kB] 2020-07-23T12:25:08.880 INFO:teuthology.orchestra.run.smithi137.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 15.2.4-377-gbdc0fb0-1bionic [45.4 kB] 2020-07-23T12:25:08.882 INFO:teuthology.orchestra.run.smithi137.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.2.4-377-gbdc0fb0-1bionic [3,157 kB] 2020-07-23T12:25:08.941 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../002-librbd1_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:09.043 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking librbd1 (15.2.4-377-gbdc0fb0-1bionic) over (12.2.13-0ubuntu0.18.04.2) ... 2020-07-23T12:25:09.222 INFO:teuthology.orchestra.run.smithi137.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.2.4-377-gbdc0fb0-1bionic [34.2 kB] 2020-07-23T12:25:09.223 INFO:teuthology.orchestra.run.smithi137.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.2.4-377-gbdc0fb0-1bionic [7,886 kB] 2020-07-23T12:25:09.579 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../003-librados2_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:09.705 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking librados2 (15.2.4-377-gbdc0fb0-1bionic) over (12.2.13-0ubuntu0.18.04.2) ... 2020-07-23T12:25:09.988 INFO:teuthology.orchestra.run.smithi137.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.2.4-377-gbdc0fb0-1bionic [3,980 B] 2020-07-23T12:25:09.989 INFO:teuthology.orchestra.run.smithi137.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.2.4-377-gbdc0fb0-1bionic [22.4 kB] 2020-07-23T12:25:09.989 INFO:teuthology.orchestra.run.smithi137.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.2.4-377-gbdc0fb0-1bionic [43.4 MB] 2020-07-23T12:25:10.359 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libcephfs2. 2020-07-23T12:25:10.372 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../004-libcephfs2_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:10.417 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libcephfs2 (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:10.694 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-rados. 2020-07-23T12:25:10.709 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../005-python3-rados_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:10.744 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-rados (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:11.089 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-07-23T12:25:11.104 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../006-python3-ceph-argparse_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:11.154 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-ceph-argparse (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:11.441 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-cephfs. 2020-07-23T12:25:11.456 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../007-python3-cephfs_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:11.499 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-cephfs (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:11.814 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-ceph-common. 2020-07-23T12:25:11.826 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../008-python3-ceph-common_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:11.861 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-ceph-common (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:12.138 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-prettytable. 2020-07-23T12:25:12.151 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-07-23T12:25:12.180 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-07-23T12:25:12.465 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-rbd. 2020-07-23T12:25:12.480 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../010-python3-rbd_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:12.526 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-rbd (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:12.886 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-07-23T12:25:12.900 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-07-23T12:25:12.943 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-07-23T12:25:13.256 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-07-23T12:25:13.260 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-07-23T12:25:13.314 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-07-23T12:25:13.361 INFO:teuthology.orchestra.run.smithi137.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.2.4-377-gbdc0fb0-1bionic [17.9 kB] 2020-07-23T12:25:13.361 INFO:teuthology.orchestra.run.smithi137.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.2.4-377-gbdc0fb0-1bionic [8,512 kB] 2020-07-23T12:25:13.623 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package librgw2. 2020-07-23T12:25:13.641 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../013-librgw2_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:13.682 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking librgw2 (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:13.910 INFO:teuthology.orchestra.run.smithi137.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/octopus/bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.2.4-377-gbdc0fb0-1bionic [62.6 kB] 2020-07-23T12:25:14.367 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-rgw. 2020-07-23T12:25:14.376 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../014-python3-rgw_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:14.426 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-rgw (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:14.631 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 157 MB in 17s (9,492 kB/s) 2020-07-23T12:25:14.738 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-07-23T12:25:14.755 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-07-23T12:25:14.776 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-uritemplate. 2020-07-23T12:25:14.786 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-07-23T12:25:14.827 INFO:teuthology.orchestra.run.smithi137.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137579 files and directories currently installed.) 2020-07-23T12:25:14.831 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-07-23T12:25:14.875 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-07-23T12:25:15.066 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-07-23T12:25:15.079 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-07-23T12:25:15.114 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-07-23T12:25:15.211 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-07-23T12:25:15.226 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-07-23T12:25:15.295 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-07-23T12:25:15.475 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package liboath0. 2020-07-23T12:25:15.489 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-07-23T12:25:15.533 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-07-23T12:25:15.654 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../002-librbd1_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:15.782 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librbd1 (15.2.4-377-gbdc0fb0-1bionic) over (12.2.13-0ubuntu0.18.04.2) ... 2020-07-23T12:25:15.853 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libradosstriper1. 2020-07-23T12:25:15.869 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../018-libradosstriper1_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:15.912 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libradosstriper1 (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:16.283 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../003-librados2_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:16.299 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-common. 2020-07-23T12:25:16.311 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../019-ceph-common_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:16.339 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-common (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:16.404 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librados2 (15.2.4-377-gbdc0fb0-1bionic) over (12.2.13-0ubuntu0.18.04.2) ... 2020-07-23T12:25:17.040 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libcephfs2. 2020-07-23T12:25:17.054 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../004-libcephfs2_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:17.098 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libcephfs2 (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:17.375 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rados. 2020-07-23T12:25:17.398 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../005-python3-rados_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:17.425 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rados (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:17.969 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-07-23T12:25:17.983 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../006-python3-ceph-argparse_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:18.028 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-ceph-argparse (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:18.086 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-base. 2020-07-23T12:25:18.100 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../020-ceph-base_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:18.154 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-base (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:18.322 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cephfs. 2020-07-23T12:25:18.341 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../007-python3-cephfs_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:18.372 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cephfs (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:18.693 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-ceph-common. 2020-07-23T12:25:18.707 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../008-python3-ceph-common_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:18.750 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-ceph-common (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:18.978 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-dateutil. 2020-07-23T12:25:18.990 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-07-23T12:25:19.019 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-07-23T12:25:19.027 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-prettytable. 2020-07-23T12:25:19.042 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-07-23T12:25:19.078 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-07-23T12:25:19.338 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rbd. 2020-07-23T12:25:19.353 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-asn1crypto. 2020-07-23T12:25:19.355 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../010-python3-rbd_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:19.366 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-07-23T12:25:19.397 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rbd (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:19.402 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-07-23T12:25:19.688 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-cffi-backend. 2020-07-23T12:25:19.702 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-07-23T12:25:19.746 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-07-23T12:25:19.766 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-07-23T12:25:19.780 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-07-23T12:25:19.825 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-07-23T12:25:20.065 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-cryptography. 2020-07-23T12:25:20.082 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... 2020-07-23T12:25:20.129 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-07-23T12:25:20.130 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-07-23T12:25:20.142 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-07-23T12:25:20.177 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-07-23T12:25:20.480 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librgw2. 2020-07-23T12:25:20.503 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../013-librgw2_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:20.528 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-openssl. 2020-07-23T12:25:20.534 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-07-23T12:25:20.538 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librgw2 (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:20.578 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-07-23T12:25:20.838 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-07-23T12:25:20.852 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:20.896 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-modules-core (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:21.199 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rgw. 2020-07-23T12:25:21.215 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../014-python3-rgw_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:21.257 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rgw (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:21.275 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-bcrypt. 2020-07-23T12:25:21.289 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-07-23T12:25:21.333 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-07-23T12:25:21.585 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-07-23T12:25:21.602 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-07-23T12:25:21.643 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-07-23T12:25:21.644 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-cherrypy3. 2020-07-23T12:25:21.669 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-07-23T12:25:21.694 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-07-23T12:25:21.937 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-07-23T12:25:21.943 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-07-23T12:25:21.979 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-07-23T12:25:22.056 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-jwt. 2020-07-23T12:25:22.069 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-07-23T12:25:22.116 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-07-23T12:25:22.357 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package liboath0. 2020-07-23T12:25:22.371 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-07-23T12:25:22.416 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-07-23T12:25:22.426 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-logutils. 2020-07-23T12:25:22.442 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-07-23T12:25:22.483 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-07-23T12:25:22.743 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libradosstriper1. 2020-07-23T12:25:22.752 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../018-libradosstriper1_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:22.793 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libradosstriper1 (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:23.103 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-markupsafe. 2020-07-23T12:25:23.119 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-07-23T12:25:23.146 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-common. 2020-07-23T12:25:23.161 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-07-23T12:25:23.162 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../019-ceph-common_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:23.196 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-common (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:23.464 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-mako. 2020-07-23T12:25:23.477 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-07-23T12:25:23.522 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-07-23T12:25:23.875 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-simplegeneric. 2020-07-23T12:25:23.888 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-07-23T12:25:23.933 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-07-23T12:25:24.246 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-singledispatch. 2020-07-23T12:25:24.258 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-07-23T12:25:24.296 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-07-23T12:25:24.638 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-webob. 2020-07-23T12:25:24.653 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-07-23T12:25:24.703 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-07-23T12:25:24.952 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-base. 2020-07-23T12:25:24.967 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../020-ceph-base_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:25.016 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-bs4. 2020-07-23T12:25:25.020 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-base (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:25.030 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-07-23T12:25:25.074 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-07-23T12:25:25.394 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-waitress. 2020-07-23T12:25:25.408 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-07-23T12:25:25.452 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-07-23T12:25:25.780 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-tempita. 2020-07-23T12:25:25.793 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-07-23T12:25:25.839 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-07-23T12:25:25.873 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-dateutil. 2020-07-23T12:25:25.887 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-07-23T12:25:25.932 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-07-23T12:25:26.151 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-paste. 2020-07-23T12:25:26.164 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-07-23T12:25:26.208 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-07-23T12:25:26.280 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-asn1crypto. 2020-07-23T12:25:26.294 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-07-23T12:25:26.334 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-07-23T12:25:26.519 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-07-23T12:25:26.534 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-07-23T12:25:26.579 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-07-23T12:25:26.662 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cffi-backend. 2020-07-23T12:25:26.676 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-07-23T12:25:26.721 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-07-23T12:25:26.882 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pastedeploy. 2020-07-23T12:25:26.894 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-07-23T12:25:26.930 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-07-23T12:25:27.031 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cryptography. 2020-07-23T12:25:27.046 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... 2020-07-23T12:25:27.084 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-07-23T12:25:27.232 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-webtest. 2020-07-23T12:25:27.249 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-07-23T12:25:27.295 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-07-23T12:25:27.535 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-openssl. 2020-07-23T12:25:27.552 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-07-23T12:25:27.577 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pecan. 2020-07-23T12:25:27.591 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-07-23T12:25:27.593 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-07-23T12:25:27.619 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-07-23T12:25:28.504 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-07-23T12:25:28.519 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:28.572 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-modules-core (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:28.732 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-werkzeug. 2020-07-23T12:25:28.745 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-07-23T12:25:28.790 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-07-23T12:25:28.858 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-bcrypt. 2020-07-23T12:25:28.872 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-07-23T12:25:28.903 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-07-23T12:25:29.186 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr. 2020-07-23T12:25:29.202 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../045-ceph-mgr_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:29.228 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cherrypy3. 2020-07-23T12:25:29.243 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-07-23T12:25:29.244 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:29.286 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-07-23T12:25:29.623 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jwt. 2020-07-23T12:25:29.638 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-07-23T12:25:29.694 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mon. 2020-07-23T12:25:29.695 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-07-23T12:25:29.705 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../046-ceph-mon_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:29.746 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mon (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:29.996 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-logutils. 2020-07-23T12:25:30.007 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-07-23T12:25:30.051 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-07-23T12:25:30.387 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-markupsafe. 2020-07-23T12:25:30.400 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-07-23T12:25:30.437 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-07-23T12:25:30.649 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-osd. 2020-07-23T12:25:30.667 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../047-ceph-osd_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:30.707 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-osd (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:30.748 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-mako. 2020-07-23T12:25:30.762 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-07-23T12:25:30.806 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-07-23T12:25:31.101 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-simplegeneric. 2020-07-23T12:25:31.116 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-07-23T12:25:31.159 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-07-23T12:25:31.463 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-singledispatch. 2020-07-23T12:25:31.475 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-07-23T12:25:31.520 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-07-23T12:25:31.858 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-webob. 2020-07-23T12:25:31.871 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-07-23T12:25:31.939 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-07-23T12:25:32.268 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-bs4. 2020-07-23T12:25:32.282 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-07-23T12:25:32.325 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-07-23T12:25:32.620 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-waitress. 2020-07-23T12:25:32.636 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-07-23T12:25:32.655 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph. 2020-07-23T12:25:32.669 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../048-ceph_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:32.678 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-07-23T12:25:32.717 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:32.981 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-tempita. 2020-07-23T12:25:32.998 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-07-23T12:25:33.058 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-fuse. 2020-07-23T12:25:33.059 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-07-23T12:25:33.064 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../049-ceph-fuse_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:33.107 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-fuse (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:33.351 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-paste. 2020-07-23T12:25:33.368 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-07-23T12:25:33.402 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-07-23T12:25:33.528 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mds. 2020-07-23T12:25:33.542 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../050-ceph-mds_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:33.586 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mds (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:33.695 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-07-23T12:25:33.711 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-07-23T12:25:33.753 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-07-23T12:25:34.039 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pastedeploy. 2020-07-23T12:25:34.056 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-07-23T12:25:34.088 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package cephadm. 2020-07-23T12:25:34.097 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-07-23T12:25:34.104 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../051-cephadm_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:34.140 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking cephadm (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:34.394 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-jinja2. 2020-07-23T12:25:34.409 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-07-23T12:25:34.410 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-webtest. 2020-07-23T12:25:34.422 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-07-23T12:25:34.449 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-07-23T12:25:34.459 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-07-23T12:25:34.738 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-07-23T12:25:34.755 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:34.795 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-cephadm (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:34.817 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pecan. 2020-07-23T12:25:34.827 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-07-23T12:25:34.870 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-07-23T12:25:35.114 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-repoze.lru. 2020-07-23T12:25:35.128 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-07-23T12:25:35.172 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-07-23T12:25:35.234 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-werkzeug. 2020-07-23T12:25:35.247 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-07-23T12:25:35.294 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-07-23T12:25:35.458 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-routes. 2020-07-23T12:25:35.474 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-07-23T12:25:35.516 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-07-23T12:25:35.679 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr. 2020-07-23T12:25:35.695 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../045-ceph-mgr_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:35.736 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:35.857 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-07-23T12:25:35.873 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:35.910 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-dashboard (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:36.172 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mon. 2020-07-23T12:25:36.188 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../046-ceph-mon_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:36.231 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mon (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:36.834 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2020-07-23T12:25:36.848 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../057-ceph-mgr-diskprediction-cloud_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:36.893 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:37.143 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-osd. 2020-07-23T12:25:37.158 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../047-ceph-osd_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:37.201 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-osd (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:37.212 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-numpy. 2020-07-23T12:25:37.225 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-07-23T12:25:37.266 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-07-23T12:25:37.875 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-decorator. 2020-07-23T12:25:37.890 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2020-07-23T12:25:37.942 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-07-23T12:25:38.270 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-scipy. 2020-07-23T12:25:38.283 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-07-23T12:25:38.328 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-07-23T12:25:39.110 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph. 2020-07-23T12:25:39.122 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../048-ceph_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:39.165 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:39.485 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-fuse. 2020-07-23T12:25:39.497 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../049-ceph-fuse_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:39.534 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-fuse (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:39.629 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-07-23T12:25:39.644 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-07-23T12:25:39.687 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-07-23T12:25:39.947 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mds. 2020-07-23T12:25:39.966 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../050-ceph-mds_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:40.006 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mds (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:40.132 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-joblib. 2020-07-23T12:25:40.148 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2020-07-23T12:25:40.190 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-joblib (0.11-1) ... 2020-07-23T12:25:40.543 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-sklearn. 2020-07-23T12:25:40.559 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2020-07-23T12:25:40.559 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package cephadm. 2020-07-23T12:25:40.563 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../051-cephadm_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:40.607 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking cephadm (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:40.610 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-07-23T12:25:40.927 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jinja2. 2020-07-23T12:25:40.940 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-07-23T12:25:40.977 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-07-23T12:25:41.166 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-07-23T12:25:41.182 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:41.224 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:41.296 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-07-23T12:25:41.311 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:41.354 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-cephadm (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:41.674 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-repoze.lru. 2020-07-23T12:25:41.690 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-07-23T12:25:41.733 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-07-23T12:25:42.069 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-routes. 2020-07-23T12:25:42.083 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-07-23T12:25:42.127 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-07-23T12:25:42.316 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-monotonic. 2020-07-23T12:25:42.325 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2020-07-23T12:25:42.369 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-07-23T12:25:42.447 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-07-23T12:25:42.462 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:42.505 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-dashboard (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:42.680 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-fasteners. 2020-07-23T12:25:42.695 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2020-07-23T12:25:42.738 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-07-23T12:25:43.099 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-httplib2. 2020-07-23T12:25:43.115 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-07-23T12:25:43.158 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-07-23T12:25:43.445 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2020-07-23T12:25:43.460 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../057-ceph-mgr-diskprediction-cloud_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:43.490 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pyasn1. 2020-07-23T12:25:43.502 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2020-07-23T12:25:43.506 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:43.535 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-07-23T12:25:43.818 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-numpy. 2020-07-23T12:25:43.833 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-07-23T12:25:43.863 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-07-23T12:25:43.873 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-07-23T12:25:43.877 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-07-23T12:25:43.922 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-07-23T12:25:44.233 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-rsa. 2020-07-23T12:25:44.247 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2020-07-23T12:25:44.291 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-07-23T12:25:44.462 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-decorator. 2020-07-23T12:25:44.476 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2020-07-23T12:25:44.521 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-07-23T12:25:44.586 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-07-23T12:25:44.602 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-07-23T12:25:44.646 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-07-23T12:25:44.848 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-scipy. 2020-07-23T12:25:44.862 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-07-23T12:25:44.919 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-07-23T12:25:45.065 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-oauth2client. 2020-07-23T12:25:45.082 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2020-07-23T12:25:45.123 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-07-23T12:25:45.461 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-setuptools. 2020-07-23T12:25:45.474 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2020-07-23T12:25:45.517 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-07-23T12:25:45.921 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-websocket. 2020-07-23T12:25:45.936 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-07-23T12:25:45.979 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-07-23T12:25:46.057 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-07-23T12:25:46.076 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-07-23T12:25:46.115 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-07-23T12:25:46.307 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-kubernetes. 2020-07-23T12:25:46.323 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-07-23T12:25:46.365 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-07-23T12:25:46.569 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-joblib. 2020-07-23T12:25:46.583 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2020-07-23T12:25:46.627 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-joblib (0.11-1) ... 2020-07-23T12:25:46.805 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-07-23T12:25:46.821 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:46.863 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-k8sevents (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:46.955 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-sklearn. 2020-07-23T12:25:46.972 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2020-07-23T12:25:47.014 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-07-23T12:25:47.186 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-json-pointer. 2020-07-23T12:25:47.197 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2020-07-23T12:25:47.240 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-07-23T12:25:47.528 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-07-23T12:25:47.545 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:47.586 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:47.602 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-jsonpatch. 2020-07-23T12:25:47.615 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-07-23T12:25:47.663 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-07-23T12:25:47.971 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-07-23T12:25:47.984 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../079-ceph-mgr-rook_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:48.021 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-mgr-rook (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:48.349 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libonig4:amd64. 2020-07-23T12:25:48.370 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2020-07-23T12:25:48.485 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-07-23T12:25:48.838 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libjq1:amd64. 2020-07-23T12:25:48.853 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2020-07-23T12:25:48.896 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-07-23T12:25:49.123 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-monotonic. 2020-07-23T12:25:49.137 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2020-07-23T12:25:49.182 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package jq. 2020-07-23T12:25:49.189 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-07-23T12:25:49.196 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2020-07-23T12:25:49.240 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-07-23T12:25:49.475 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-fasteners. 2020-07-23T12:25:49.492 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2020-07-23T12:25:49.493 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package socat. 2020-07-23T12:25:49.508 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-07-23T12:25:49.534 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-07-23T12:25:49.551 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-07-23T12:25:49.825 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-httplib2. 2020-07-23T12:25:49.840 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-07-23T12:25:49.878 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-07-23T12:25:49.930 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-07-23T12:25:49.947 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-07-23T12:25:49.988 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-07-23T12:25:50.189 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pyasn1. 2020-07-23T12:25:50.204 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2020-07-23T12:25:50.247 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-07-23T12:25:50.282 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package xmlstarlet. 2020-07-23T12:25:50.297 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2020-07-23T12:25:50.341 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-07-23T12:25:50.590 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-07-23T12:25:50.602 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-07-23T12:25:50.644 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-07-23T12:25:50.711 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package ceph-test. 2020-07-23T12:25:50.725 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../086-ceph-test_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:50.780 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking ceph-test (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:50.966 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rsa. 2020-07-23T12:25:50.983 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2020-07-23T12:25:51.023 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-07-23T12:25:51.334 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-07-23T12:25:51.348 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-07-23T12:25:51.392 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-07-23T12:25:51.873 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-oauth2client. 2020-07-23T12:25:51.887 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2020-07-23T12:25:51.930 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-07-23T12:25:52.283 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-setuptools. 2020-07-23T12:25:52.297 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2020-07-23T12:25:52.341 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-07-23T12:25:52.761 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-websocket. 2020-07-23T12:25:52.775 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-07-23T12:25:52.824 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-07-23T12:25:53.131 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-kubernetes. 2020-07-23T12:25:53.146 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-07-23T12:25:53.193 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-07-23T12:25:53.635 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-07-23T12:25:53.650 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:53.693 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-k8sevents (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:54.055 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-json-pointer. 2020-07-23T12:25:54.070 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2020-07-23T12:25:54.112 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-07-23T12:25:54.190 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package libcephfs-dev. 2020-07-23T12:25:54.205 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../087-libcephfs-dev_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:54.248 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking libcephfs-dev (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:54.490 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jsonpatch. 2020-07-23T12:25:54.515 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-07-23T12:25:54.551 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-07-23T12:25:54.559 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python-attr. 2020-07-23T12:25:54.573 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-07-23T12:25:54.618 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python-attr (17.4.0-2) ... 2020-07-23T12:25:54.854 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-07-23T12:25:54.865 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../079-ceph-mgr-rook_15.2.4-377-gbdc0fb0-1bionic_all.deb ... 2020-07-23T12:25:54.910 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-rook (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:54.920 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python-funcsigs. 2020-07-23T12:25:54.936 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-07-23T12:25:54.996 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-07-23T12:25:55.238 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libonig4:amd64. 2020-07-23T12:25:55.252 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2020-07-23T12:25:55.298 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python-pluggy. 2020-07-23T12:25:55.312 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-07-23T12:25:55.356 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-07-23T12:25:55.381 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-07-23T12:25:55.668 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python-py. 2020-07-23T12:25:55.684 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-07-23T12:25:55.726 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python-py (1.5.2-1) ... 2020-07-23T12:25:55.735 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libjq1:amd64. 2020-07-23T12:25:55.750 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2020-07-23T12:25:55.803 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-07-23T12:25:56.049 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python-pytest. 2020-07-23T12:25:56.053 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package jq. 2020-07-23T12:25:56.065 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-07-23T12:25:56.070 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2020-07-23T12:25:56.112 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-07-23T12:25:56.113 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-07-23T12:25:56.389 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package socat. 2020-07-23T12:25:56.403 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-07-23T12:25:56.432 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-attr. 2020-07-23T12:25:56.448 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-07-23T12:25:56.457 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-07-23T12:25:56.490 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-07-23T12:25:56.793 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-cycler. 2020-07-23T12:25:56.808 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-07-23T12:25:56.843 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-07-23T12:25:56.861 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-07-23T12:25:56.870 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-07-23T12:25:56.914 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-07-23T12:25:57.137 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-webencodings. 2020-07-23T12:25:57.153 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-07-23T12:25:57.196 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-07-23T12:25:57.196 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package xmlstarlet. 2020-07-23T12:25:57.210 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2020-07-23T12:25:57.282 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-07-23T12:25:57.448 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-html5lib. 2020-07-23T12:25:57.462 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-07-23T12:25:57.499 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-07-23T12:25:57.616 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-test. 2020-07-23T12:25:57.630 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../086-ceph-test_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:25:57.674 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-test (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:25:57.853 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-07-23T12:25:57.867 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-07-23T12:25:57.910 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-07-23T12:25:58.350 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pyparsing. 2020-07-23T12:25:58.362 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-07-23T12:25:58.405 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-07-23T12:25:58.724 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-tz. 2020-07-23T12:25:58.737 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-07-23T12:25:58.782 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-tz (2018.3-2) ... 2020-07-23T12:25:59.069 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-matplotlib. 2020-07-23T12:25:59.084 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-07-23T12:25:59.127 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-07-23T12:25:59.832 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-olefile. 2020-07-23T12:25:59.847 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-07-23T12:25:59.890 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-07-23T12:26:00.218 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pastescript. 2020-07-23T12:26:00.234 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-07-23T12:26:00.276 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-07-23T12:26:00.646 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pil:amd64. 2020-07-23T12:26:00.661 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-07-23T12:26:00.704 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-07-23T12:26:00.970 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libcephfs-dev. 2020-07-23T12:26:00.986 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../087-libcephfs-dev_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:26:01.028 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libcephfs-dev (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:01.074 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pluggy. 2020-07-23T12:26:01.089 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-07-23T12:26:01.132 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-07-23T12:26:01.357 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python-attr. 2020-07-23T12:26:01.363 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-07-23T12:26:01.406 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python-attr (17.4.0-2) ... 2020-07-23T12:26:01.460 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-py. 2020-07-23T12:26:01.476 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-07-23T12:26:01.518 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-py (1.5.2-1) ... 2020-07-23T12:26:01.709 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python-funcsigs. 2020-07-23T12:26:01.723 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-07-23T12:26:01.767 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-07-23T12:26:01.889 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pyinotify. 2020-07-23T12:26:01.904 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-07-23T12:26:01.947 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-07-23T12:26:02.070 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python-pluggy. 2020-07-23T12:26:02.084 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-07-23T12:26:02.128 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-07-23T12:26:02.243 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-pytest. 2020-07-23T12:26:02.258 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-07-23T12:26:02.299 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-07-23T12:26:02.431 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python-py. 2020-07-23T12:26:02.454 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-07-23T12:26:02.489 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python-py (1.5.2-1) ... 2020-07-23T12:26:02.628 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-simplejson. 2020-07-23T12:26:02.645 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-07-23T12:26:02.686 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-07-23T12:26:02.809 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python-pytest. 2020-07-23T12:26:02.823 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-07-23T12:26:02.887 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-07-23T12:26:03.023 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package python3-tk:amd64. 2020-07-23T12:26:03.037 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-07-23T12:26:03.081 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-07-23T12:26:03.229 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-attr. 2020-07-23T12:26:03.244 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-07-23T12:26:03.287 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-07-23T12:26:03.543 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package radosgw. 2020-07-23T12:26:03.557 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../110-radosgw_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:26:03.590 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cycler. 2020-07-23T12:26:03.601 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking radosgw (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:03.606 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-07-23T12:26:03.648 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-07-23T12:26:03.951 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-webencodings. 2020-07-23T12:26:03.967 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-07-23T12:26:04.009 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-07-23T12:26:04.336 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-html5lib. 2020-07-23T12:26:04.346 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-07-23T12:26:04.387 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-07-23T12:26:04.522 INFO:teuthology.orchestra.run.smithi133.stdout:Selecting previously unselected package rbd-fuse. 2020-07-23T12:26:04.535 INFO:teuthology.orchestra.run.smithi133.stdout:Preparing to unpack .../111-rbd-fuse_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:26:04.580 INFO:teuthology.orchestra.run.smithi133.stdout:Unpacking rbd-fuse (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:04.732 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-07-23T12:26:04.754 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-07-23T12:26:04.790 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-07-23T12:26:04.894 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-07-23T12:26:05.189 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-07-23T12:26:05.235 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pyparsing. 2020-07-23T12:26:05.254 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-07-23T12:26:05.293 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-07-23T12:26:05.562 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-07-23T12:26:05.604 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-tz. 2020-07-23T12:26:05.622 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-07-23T12:26:05.663 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-tz (2018.3-2) ... 2020-07-23T12:26:05.688 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-07-23T12:26:06.002 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-matplotlib. 2020-07-23T12:26:06.014 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-07-23T12:26:06.057 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-07-23T12:26:06.299 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-07-23T12:26:06.683 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-07-23T12:26:06.738 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-olefile. 2020-07-23T12:26:06.752 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-07-23T12:26:06.802 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-07-23T12:26:06.951 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-webencodings (0.5-2) ... 2020-07-23T12:26:07.090 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pastescript. 2020-07-23T12:26:07.107 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-07-23T12:26:07.141 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-07-23T12:26:07.248 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python-attr (17.4.0-2) ... 2020-07-23T12:26:07.438 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-07-23T12:26:07.511 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pil:amd64. 2020-07-23T12:26:07.526 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-07-23T12:26:07.573 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-py (1.5.2-1) ... 2020-07-23T12:26:07.575 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-07-23T12:26:07.898 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-07-23T12:26:07.948 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pluggy. 2020-07-23T12:26:07.963 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-07-23T12:26:08.006 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-07-23T12:26:08.092 INFO:teuthology.orchestra.run.smithi133.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-07-23T12:26:08.325 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-py. 2020-07-23T12:26:08.340 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-07-23T12:26:08.384 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-py (1.5.2-1) ... 2020-07-23T12:26:08.396 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-07-23T12:26:08.755 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pyinotify. 2020-07-23T12:26:08.756 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-ceph-argparse (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:08.776 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-07-23T12:26:08.813 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-07-23T12:26:09.061 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-07-23T12:26:09.240 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pytest. 2020-07-23T12:26:09.255 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-07-23T12:26:09.298 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-07-23T12:26:09.363 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-07-23T12:26:09.489 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-07-23T12:26:09.619 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-simplejson. 2020-07-23T12:26:09.636 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-07-23T12:26:09.677 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-07-23T12:26:09.982 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-tk:amd64. 2020-07-23T12:26:09.988 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-07-23T12:26:10.030 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-07-23T12:26:10.416 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-07-23T12:26:10.467 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package radosgw. 2020-07-23T12:26:10.483 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../110-radosgw_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:26:10.525 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking radosgw (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:10.718 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-07-23T12:26:11.002 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python-py (1.5.2-1) ... 2020-07-23T12:26:11.220 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-07-23T12:26:11.462 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package rbd-fuse. 2020-07-23T12:26:11.479 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../111-rbd-fuse_15.2.4-377-gbdc0fb0-1bionic_amd64.deb ... 2020-07-23T12:26:11.520 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking rbd-fuse (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:11.555 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-07-23T12:26:11.645 INFO:teuthology.orchestra.run.smithi133.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-07-23T12:26:11.823 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-07-23T12:26:11.907 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-07-23T12:26:12.049 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-07-23T12:26:12.133 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-07-23T12:26:12.250 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-07-23T12:26:12.485 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-07-23T12:26:12.518 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-07-23T12:26:12.616 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-07-23T12:26:12.844 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-07-23T12:26:13.162 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-07-23T12:26:13.213 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-07-23T12:26:13.522 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-07-23T12:26:13.581 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-07-23T12:26:13.840 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-07-23T12:26:13.849 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-webencodings (0.5-2) ... 2020-07-23T12:26:14.133 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up cephadm (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:14.154 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python-attr (17.4.0-2) ... 2020-07-23T12:26:14.369 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-07-23T12:26:14.497 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-py (1.5.2-1) ... 2020-07-23T12:26:14.801 INFO:teuthology.orchestra.run.smithi133.stdout:Adding system user cephadm....done 2020-07-23T12:26:14.838 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-07-23T12:26:14.966 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-07-23T12:26:15.033 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-07-23T12:26:15.266 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-07-23T12:26:15.324 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-07-23T12:26:15.392 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-07-23T12:26:15.617 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-ceph-argparse (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:15.727 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-07-23T12:26:15.918 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-07-23T12:26:16.011 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-07-23T12:26:16.137 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-07-23T12:26:16.220 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-07-23T12:26:16.345 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-07-23T12:26:16.473 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-ceph-common (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:16.801 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-monotonic (1.1-2) ... 2020-07-23T12:26:17.083 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-07-23T12:26:17.209 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-07-23T12:26:17.274 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-07-23T12:26:17.560 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-07-23T12:26:17.591 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-07-23T12:26:17.886 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python-py (1.5.2-1) ... 2020-07-23T12:26:17.912 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-07-23T12:26:18.001 INFO:teuthology.orchestra.run.smithi133.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-07-23T12:26:18.038 INFO:teuthology.orchestra.run.smithi133.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-07-23T12:26:18.111 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-07-23T12:26:18.273 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-07-23T12:26:18.445 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-07-23T12:26:18.535 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-07-23T12:26:18.565 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-07-23T12:26:18.780 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-07-23T12:26:18.866 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-07-23T12:26:18.906 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-07-23T12:26:19.099 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-07-23T12:26:19.200 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-attr (17.4.0-2) ... 2020-07-23T12:26:19.493 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-07-23T12:26:19.650 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-07-23T12:26:19.673 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-07-23T12:26:19.989 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-07-23T12:26:20.088 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-07-23T12:26:20.177 INFO:teuthology.orchestra.run.smithi133.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-07-23T12:26:20.286 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-07-23T12:26:20.423 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up liboath0 (2.6.1-1) ... 2020-07-23T12:26:20.599 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-07-23T12:26:20.662 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-07-23T12:26:20.725 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-07-23T12:26:20.968 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-07-23T12:26:20.997 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-07-23T12:26:21.081 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-07-23T12:26:21.298 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up cephadm (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:21.456 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-07-23T12:26:21.581 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-07-23T12:26:21.865 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-tz (2018.3-2) ... 2020-07-23T12:26:21.990 INFO:teuthology.orchestra.run.smithi137.stdout:Adding system user cephadm....done 2020-07-23T12:26:22.110 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-07-23T12:26:22.157 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-07-23T12:26:22.406 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-07-23T12:26:22.458 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-07-23T12:26:22.533 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-07-23T12:26:22.752 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-joblib (0.11-1) ... 2020-07-23T12:26:22.876 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-07-23T12:26:23.111 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-routes (2.4.1-1) ... 2020-07-23T12:26:23.176 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-07-23T12:26:23.310 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-07-23T12:26:23.438 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-07-23T12:26:23.628 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-ceph-common (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:23.742 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-07-23T12:26:23.921 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-monotonic (1.1-2) ... 2020-07-23T12:26:24.065 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-07-23T12:26:24.213 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-07-23T12:26:24.339 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-07-23T12:26:24.438 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-07-23T12:26:24.707 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-07-23T12:26:24.751 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up jq (1.5+dfsg-2) ... 2020-07-23T12:26:24.869 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-07-23T12:26:25.067 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-07-23T12:26:25.157 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-07-23T12:26:25.211 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-07-23T12:26:25.461 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-07-23T12:26:25.745 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-07-23T12:26:26.031 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-07-23T12:26:26.031 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-07-23T12:26:26.334 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-07-23T12:26:26.364 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-attr (17.4.0-2) ... 2020-07-23T12:26:26.660 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-07-23T12:26:26.665 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-07-23T12:26:26.769 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-07-23T12:26:26.866 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-07-23T12:26:27.137 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python-pytest (3.3.2-2) ... 2020-07-23T12:26:27.276 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-07-23T12:26:27.366 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-07-23T12:26:27.389 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-07-23T12:26:27.515 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-07-23T12:26:27.620 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up liboath0 (2.6.1-1) ... 2020-07-23T12:26:27.641 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up librados2 (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:27.767 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-07-23T12:26:27.779 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-07-23T12:26:27.897 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-07-23T12:26:28.085 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libcephfs2 (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:28.148 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-07-23T12:26:28.211 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libcephfs-dev (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:28.299 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-07-23T12:26:28.337 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-rados (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:28.471 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-07-23T12:26:28.701 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-07-23T12:26:28.791 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-07-23T12:26:28.835 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-07-23T12:26:29.136 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-tz (2018.3-2) ... 2020-07-23T12:26:29.351 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-07-23T12:26:29.413 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-07-23T12:26:29.688 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-07-23T12:26:29.975 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-07-23T12:26:30.174 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-joblib (0.11-1) ... 2020-07-23T12:26:30.349 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-routes (2.4.1-1) ... 2020-07-23T12:26:30.642 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-07-23T12:26:30.829 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-07-23T12:26:30.960 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-07-23T12:26:31.303 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-07-23T12:26:31.654 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-07-23T12:26:31.734 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-07-23T12:26:31.930 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up jq (1.5+dfsg-2) ... 2020-07-23T12:26:32.026 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up libradosstriper1 (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:32.056 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-07-23T12:26:32.136 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-cephfs (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:32.437 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up librgw2 (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:32.563 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-fuse (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:32.781 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-07-23T12:26:32.782 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-07-23T12:26:33.048 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up librbd1 (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:33.174 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-07-23T12:26:33.220 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-07-23T12:26:33.264 INFO:teuthology.orchestra.run.smithi133.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-07-23T12:26:33.304 INFO:teuthology.orchestra.run.smithi133.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-07-23T12:26:33.521 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-07-23T12:26:33.602 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-07-23T12:26:33.839 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-07-23T12:26:33.982 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-07-23T12:26:34.247 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-modules-core (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:34.350 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python-pytest (3.3.2-2) ... 2020-07-23T12:26:34.373 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-07-23T12:26:34.614 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-07-23T12:26:34.716 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-rgw (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:34.737 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-07-23T12:26:34.844 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up rbd-fuse (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:34.853 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librados2 (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:34.954 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-07-23T12:26:34.968 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up python3-rbd (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:35.094 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-common (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:35.247 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libcephfs2 (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:35.448 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libcephfs-dev (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:35.574 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rados (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:35.619 INFO:teuthology.orchestra.run.smithi133.stdout:Adding group ceph....done 2020-07-23T12:26:35.700 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-07-23T12:26:36.008 INFO:teuthology.orchestra.run.smithi133.stdout:Adding system user ceph....done 2020-07-23T12:26:36.034 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-07-23T12:26:36.219 INFO:teuthology.orchestra.run.smithi133.stdout:Setting system user ceph properties....done 2020-07-23T12:26:36.353 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-07-23T12:26:36.380 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-07-23T12:26:36.606 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-07-23T12:26:36.695 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-07-23T12:26:36.873 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up radosgw (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:37.128 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-07-23T12:26:37.129 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-07-23T12:26:37.508 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-test (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:37.531 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-07-23T12:26:37.634 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-base (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:37.963 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-07-23T12:26:38.231 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:38.434 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-07-23T12:26:38.434 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-07-23T12:26:38.440 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-07-23T12:26:38.685 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-diskprediction-local (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:38.741 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libradosstriper1 (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:38.821 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-osd (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:38.870 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cephfs (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:39.159 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librgw2 (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:39.271 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-07-23T12:26:39.271 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-07-23T12:26:39.285 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-fuse (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:39.505 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-07-23T12:26:39.506 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-07-23T12:26:39.532 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-cephadm (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:39.787 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librbd1 (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:39.809 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:39.888 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-07-23T12:26:39.952 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-rook (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:39.970 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-07-23T12:26:40.010 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-07-23T12:26:40.079 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mds (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:40.295 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-07-23T12:26:40.296 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-07-23T12:26:40.308 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-07-23T12:26:40.558 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mon (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:40.808 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-07-23T12:26:40.808 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-07-23T12:26:40.994 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-modules-core (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:41.066 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-k8sevents (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:41.120 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-07-23T12:26:41.191 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph-mgr-dashboard (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:41.338 INFO:teuthology.orchestra.run.smithi133.stdout:Setting up ceph (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:41.421 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rgw (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:41.459 INFO:teuthology.orchestra.run.smithi133.stdout:Processing triggers for systemd (237-3ubuntu10.41) ... 2020-07-23T12:26:41.530 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up rbd-fuse (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:41.644 INFO:teuthology.orchestra.run.smithi133.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-07-23T12:26:41.662 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rbd (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:41.785 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-common (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:42.290 INFO:teuthology.orchestra.run.smithi137.stdout:Adding group ceph....done 2020-07-23T12:26:42.636 INFO:teuthology.orchestra.run.smithi137.stdout:Adding system user ceph....done 2020-07-23T12:26:42.836 INFO:teuthology.orchestra.run.smithi137.stdout:Setting system user ceph properties....done 2020-07-23T12:26:42.879 INFO:teuthology.orchestra.run.smithi133.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-07-23T12:26:42.921 INFO:teuthology.orchestra.run.smithi133.stdout:ureadahead will be reprofiled on next reboot 2020-07-23T12:26:42.979 INFO:teuthology.orchestra.run.smithi133.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.2) ... 2020-07-23T12:26:42.997 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-07-23T12:26:43.223 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-07-23T12:26:43.434 INFO:teuthology.orchestra.run.smithi133.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-07-23T12:26:43.436 DEBUG:teuthology.parallel:result is None 2020-07-23T12:26:43.518 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up radosgw (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:43.782 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-07-23T12:26:43.782 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-07-23T12:26:44.186 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-test (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:44.287 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-base (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:44.587 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-07-23T12:26:44.981 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:45.203 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-07-23T12:26:45.204 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-07-23T12:26:45.474 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-diskprediction-local (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:45.617 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-osd (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:46.068 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-07-23T12:26:46.068 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-07-23T12:26:46.344 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-cephadm (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:46.470 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:46.588 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-rook (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:46.706 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mds (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:46.930 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-07-23T12:26:46.931 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-07-23T12:26:47.207 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mon (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:47.416 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-07-23T12:26:47.417 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-07-23T12:26:47.700 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-k8sevents (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:47.826 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-dashboard (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:47.985 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph (15.2.4-377-gbdc0fb0-1bionic) ... 2020-07-23T12:26:48.112 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for systemd (237-3ubuntu10.41) ... 2020-07-23T12:26:48.321 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-07-23T12:26:49.565 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-07-23T12:26:49.607 INFO:teuthology.orchestra.run.smithi137.stdout:ureadahead will be reprofiled on next reboot 2020-07-23T12:26:49.665 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.2) ... 2020-07-23T12:26:50.110 INFO:teuthology.orchestra.run.smithi137.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-07-23T12:26:50.111 DEBUG:teuthology.parallel:result is None 2020-07-23T12:26:50.113 INFO:teuthology.orchestra.run.smithi137:> sudo lsb_release -is 2020-07-23T12:26:50.162 INFO:teuthology.orchestra.run.smithi137.stdout:Ubuntu 2020-07-23T12:26:50.167 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-23T12:26:50.168 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac 2020-07-23T12:26:50.357 INFO:teuthology.orchestra.run.smithi137:> dpkg-query -W -f '${Version}' ceph 2020-07-23T12:26:50.382 INFO:teuthology.orchestra.run.smithi137.stdout:15.2.4-377-gbdc0fb0-1bionic 2020-07-23T12:26:50.383 INFO:teuthology.packaging:The installed version of ceph is 15.2.4-377-gbdc0fb0-1bionic 2020-07-23T12:26:50.384 INFO:teuthology.task.install:The correct ceph version 15.2.4-377-gbdc0fb0-1bionic is installed. 2020-07-23T12:26:50.384 INFO:teuthology.orchestra.run.smithi133:> sudo lsb_release -is 2020-07-23T12:26:50.439 INFO:teuthology.orchestra.run.smithi133.stdout:Ubuntu 2020-07-23T12:26:50.445 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-07-23T12:26:50.446 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac 2020-07-23T12:26:50.613 INFO:teuthology.orchestra.run.smithi133:> dpkg-query -W -f '${Version}' ceph 2020-07-23T12:26:50.636 INFO:teuthology.orchestra.run.smithi133.stdout:15.2.4-377-gbdc0fb0-1bionic 2020-07-23T12:26:50.637 INFO:teuthology.packaging:The installed version of ceph is 15.2.4-377-gbdc0fb0-1bionic 2020-07-23T12:26:50.637 INFO:teuthology.task.install:The correct ceph version 15.2.4-377-gbdc0fb0-1bionic is installed. 2020-07-23T12:26:50.638 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-07-23T12:26:50.638 INFO:teuthology.orchestra.run.smithi137:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-07-23T12:26:50.655 INFO:teuthology.orchestra.run.smithi133:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-07-23T12:26:50.692 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-07-23T12:26:50.693 INFO:teuthology.orchestra.run.smithi137:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-07-23T12:26:50.715 INFO:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-07-23T12:26:50.770 INFO:teuthology.orchestra.run.smithi133:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-07-23T12:26:50.788 INFO:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-07-23T12:26:50.844 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-07-23T12:26:50.845 INFO:teuthology.orchestra.run.smithi137:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-07-23T12:26:50.894 INFO:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-07-23T12:26:50.948 INFO:teuthology.orchestra.run.smithi133:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-07-23T12:26:50.996 INFO:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-07-23T12:26:51.050 INFO:teuthology.run_tasks:Running task ceph... 2020-07-23T12:26:51.134 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-07-23T12:26:51.135 INFO:teuthology.orchestra.run.smithi133:> sudo chmod 777 /var/log/ceph 2020-07-23T12:26:51.138 INFO:teuthology.orchestra.run.smithi137:> sudo chmod 777 /var/log/ceph 2020-07-23T12:26:51.152 INFO:tasks.ceph:Disabling ceph logrotate... 2020-07-23T12:26:51.153 INFO:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/logrotate.d/ceph 2020-07-23T12:26:51.191 INFO:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/logrotate.d/ceph 2020-07-23T12:26:51.210 INFO:tasks.ceph:Creating extra log directories... 2020-07-23T12:26:51.211 INFO:teuthology.orchestra.run.smithi133:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-07-23T12:26:51.247 INFO:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-07-23T12:26:51.267 INFO:tasks.ceph:Creating ceph cluster c1... 2020-07-23T12:26:51.268 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-07-23T12:26:51.268 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2020-07-20_05:05:02-rgw-octopus-distro-basic-smithi/5242029', 'branch': 'octopus', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '5242029', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2020-07-20_05:05:02-rgw-octopus-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'octopus'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': 'bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'octopus', 'sha1': 'bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 1000, 'repo': 'git://git.ceph.com/ceph.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': 'bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'octopus', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_octopus/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph.git', 'suite_sha1': 'bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.20760', 'targets': {'smithi137.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDujhwHQQYYd0i9/NuavIxWtoPBK11AcchJKBZRpQXB49v1LqcluhHxhXvo8P17ads2mEYs5fiU6zWmywcGE9JJ2skLImeTRfEaTwuAmAMKfGZ1Lv9jX3EHAGViMTYcpkY59w462r7A+d5gt9JY/8aNlWdNdQgEY3GA7h6iZuixavroXd28UB3wzTVLOrCCTpYDMhDlvmioE63/iXbtwUE0UYhEAsnpWk7nx8kLw+nRrHs1QPlH447Vpqpb/zx0zMeJ9e/Xvu3KyHImTa8P9eV59CW6j3FnufcbWAl/NxI6lgjXfqJQO0f4PZSlM95kS94Yp90nqD2FzaxXrQpXhJvH', 'smithi133.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDTunE7kpOkFCjYvZc7sF3Pj+bcPT2jzsHSBbPaEA/tW/bjDZRdd8emTC4L0fxv0CG02maN33JHAxJdBoLaUZuGuGaExQ0q3xlxBxslCwAn7h+uWJdYS3ZTtHDriAcNG1tK0zAy7hkF1WTJptRrQGhEMp4R+cKZqkNnrl71yMvrTCxK+RVdWdx92U4i476ggc0dQZdKst52B9oUJ6dU4fu2Ivn5GjWiYk1Voy/TyRaG8TgVEyvpKJ38UURB1Axb/P/60/Xn9WDsWp3k25z5uw3dzdEC6z3z8i8jUpHssvD0l7CGxxRZiwkt3MoKH4XY5J7cI1Ynm9QjI2YcQqqh6P8z'}} 2020-07-23T12:26:51.269 INFO:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-07-23T12:26:51.303 INFO:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-07-23T12:26:51.314 INFO:teuthology.orchestra.run.smithi133:> sudo install -d -m0777 -- /var/run/ceph 2020-07-23T12:26:51.351 INFO:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/run/ceph 2020-07-23T12:26:51.420 DEBUG:teuthology.orchestra.remote:smithi137:/scratch_devs is 72B 2020-07-23T12:26:51.430 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-07-23T12:26:51.431 INFO:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-07-23T12:26:51.437 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-07-23T12:26:51.438 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-23T12:26:51.438 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 6h/6d Inode: 342 Links: 1 2020-07-23T12:26:51.438 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-23T12:26:51.438 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2020-07-23 12:24:22.405208462 +0000 2020-07-23T12:26:51.439 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2020-07-23 12:24:22.301260465 +0000 2020-07-23T12:26:51.439 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2020-07-23 12:24:22.301260465 +0000 2020-07-23T12:26:51.439 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2020-07-23T12:26:51.445 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2020-07-23T12:26:51.446 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2020-07-23T12:26:51.446 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.00030445 s, 1.7 MB/s 2020-07-23T12:26:51.450 INFO:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-07-23T12:26:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-07-23T12:26:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-23T12:26:51.499 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 6h/6d Inode: 349 Links: 1 2020-07-23T12:26:51.499 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-23T12:26:51.499 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2020-07-23 12:24:22.525148460 +0000 2020-07-23T12:26:51.500 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2020-07-23 12:24:22.521150459 +0000 2020-07-23T12:26:51.500 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2020-07-23 12:24:22.521150459 +0000 2020-07-23T12:26:51.500 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2020-07-23T12:26:51.507 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2020-07-23T12:26:51.508 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2020-07-23T12:26:51.508 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000351204 s, 1.5 MB/s 2020-07-23T12:26:51.510 INFO:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-07-23T12:26:51.558 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-07-23T12:26:51.559 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-23T12:26:51.559 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 6h/6d Inode: 355 Links: 1 2020-07-23T12:26:51.559 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-23T12:26:51.559 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2020-07-23 12:24:22.932944450 +0000 2020-07-23T12:26:51.560 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2020-07-23 12:24:22.813004453 +0000 2020-07-23T12:26:51.560 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2020-07-23 12:24:22.813004453 +0000 2020-07-23T12:26:51.560 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2020-07-23T12:26:51.567 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2020-07-23T12:26:51.568 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2020-07-23T12:26:51.568 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000311393 s, 1.6 MB/s 2020-07-23T12:26:51.572 INFO:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-07-23T12:26:51.618 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-07-23T12:26:51.619 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-23T12:26:51.619 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 6h/6d Inode: 361 Links: 1 2020-07-23T12:26:51.619 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-23T12:26:51.619 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2020-07-23 12:24:23.052884448 +0000 2020-07-23T12:26:51.620 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2020-07-23 12:24:23.048886448 +0000 2020-07-23T12:26:51.620 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2020-07-23 12:24:23.048886448 +0000 2020-07-23T12:26:51.620 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2020-07-23T12:26:51.627 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2020-07-23T12:26:51.627 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2020-07-23T12:26:51.628 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000401367 s, 1.3 MB/s 2020-07-23T12:26:51.632 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-23T12:26:51.633 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-07-23T12:26:51.633 INFO:tasks.ceph:Generating config... 2020-07-23T12:26:51.635 INFO:tasks.ceph:[client] debug rgw = 20 2020-07-23T12:26:51.635 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-07-23T12:26:51.635 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-07-23T12:26:51.635 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-23T12:26:51.636 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-07-23T12:26:51.636 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-07-23T12:26:51.636 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-07-23T12:26:51.637 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-07-23T12:26:51.637 INFO:tasks.ceph:[client] setgroup = ceph 2020-07-23T12:26:51.637 INFO:tasks.ceph:[client] setuser = ceph 2020-07-23T12:26:51.638 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-07-23T12:26:51.638 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-07-23T12:26:51.638 INFO:tasks.ceph:[mgr] debug ms = 1 2020-07-23T12:26:51.638 INFO:tasks.ceph:[mon] debug mon = 20 2020-07-23T12:26:51.639 INFO:tasks.ceph:[mon] debug ms = 1 2020-07-23T12:26:51.639 INFO:tasks.ceph:[mon] debug paxos = 20 2020-07-23T12:26:51.639 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-07-23T12:26:51.639 INFO:tasks.ceph:[osd] debug filestore = 20 2020-07-23T12:26:51.640 INFO:tasks.ceph:[osd] debug journal = 20 2020-07-23T12:26:51.640 INFO:tasks.ceph:[osd] debug ms = 20 2020-07-23T12:26:51.640 INFO:tasks.ceph:[osd] debug osd = 25 2020-07-23T12:26:51.641 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-07-23T12:26:51.641 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-07-23T12:26:51.641 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-07-23T12:26:51.642 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-07-23T12:26:51.642 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-07-23T12:26:51.643 INFO:tasks.ceph:Setting up c1.mon.a... 2020-07-23T12:26:51.643 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2020-07-23T12:26:51.705 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/c1.keyring 2020-07-23T12:26:51.708 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2020-07-23T12:26:51.787 INFO:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-07-23T12:26:51.848 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.137')] 2020-07-23T12:26:51.848 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.137 --print /home/ubuntu/cephtest/c1.monmap 2020-07-23T12:26:51.915 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2020-07-23T12:26:51.916 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: generated fsid 2f2a5b22-0949-421c-a9cb-4c87aa049a79 2020-07-23T12:26:51.916 INFO:teuthology.orchestra.run.smithi137.stdout:epoch 0 2020-07-23T12:26:51.916 INFO:teuthology.orchestra.run.smithi137.stdout:fsid 2f2a5b22-0949-421c-a9cb-4c87aa049a79 2020-07-23T12:26:51.916 INFO:teuthology.orchestra.run.smithi137.stdout:last_changed 2020-07-23T12:26:51.917461+0000 2020-07-23T12:26:51.917 INFO:teuthology.orchestra.run.smithi137.stdout:created 2020-07-23T12:26:51.917461+0000 2020-07-23T12:26:51.917 INFO:teuthology.orchestra.run.smithi137.stdout:min_mon_release 0 (unknown) 2020-07-23T12:26:51.917 INFO:teuthology.orchestra.run.smithi137.stdout:0: [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] mon.a 2020-07-23T12:26:51.917 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2020-07-23T12:26:51.918 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 2f2a5b22-0949-421c-a9cb-4c87aa049a79... 2020-07-23T12:26:51.920 INFO:teuthology.orchestra.run.smithi133:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-07-23T12:26:51.922 INFO:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-07-23T12:26:51.963 INFO:teuthology.orchestra.run.smithi133.stdout:[global] 2020-07-23T12:26:51.963 INFO:teuthology.orchestra.run.smithi133.stdout: chdir = "" 2020-07-23T12:26:51.963 INFO:teuthology.orchestra.run.smithi133.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-07-23T12:26:51.963 INFO:teuthology.orchestra.run.smithi133.stdout: auth supported = cephx 2020-07-23T12:26:51.964 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.964 INFO:teuthology.orchestra.run.smithi133.stdout: filestore xattr use omap = true 2020-07-23T12:26:51.964 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.964 INFO:teuthology.orchestra.run.smithi133.stdout: mon clock drift allowed = 1.000 2020-07-23T12:26:51.964 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.964 INFO:teuthology.orchestra.run.smithi133.stdout: osd crush chooseleaf type = 0 2020-07-23T12:26:51.965 INFO:teuthology.orchestra.run.smithi133.stdout: auth debug = true 2020-07-23T12:26:51.965 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.965 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on old message = true 2020-07-23T12:26:51.965 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on bug = true 2020-07-23T12:26:51.965 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.966 INFO:teuthology.orchestra.run.smithi133.stdout: mon max pg per osd = 10000 # >= luminous 2020-07-23T12:26:51.966 INFO:teuthology.orchestra.run.smithi133.stdout: mon pg warn max object skew = 0 2020-07-23T12:26:51.966 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.966 INFO:teuthology.orchestra.run.smithi133.stdout: # disable pg_autoscaler by default for new pools 2020-07-23T12:26:51.967 INFO:teuthology.orchestra.run.smithi133.stdout: osd_pool_default_pg_autoscale_mode = off 2020-07-23T12:26:51.967 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.967 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default size = 2 2020-07-23T12:26:51.967 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.968 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow primary affinity = true 2020-07-23T12:26:51.968 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow pg remap = true 2020-07-23T12:26:51.968 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on legacy crush tunables = false 2020-07-23T12:26:51.968 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on crush straw calc version zero = false 2020-07-23T12:26:51.968 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on no sortbitwise = false 2020-07-23T12:26:51.969 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on osd down out interval zero = false 2020-07-23T12:26:51.969 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on too few osds = false 2020-07-23T12:26:51.969 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-07-23T12:26:51.969 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_no_redundancy = false 2020-07-23T12:26:51.969 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.970 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-07-23T12:26:51.970 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.970 INFO:teuthology.orchestra.run.smithi133.stdout: osd default data pool replay window = 5 2020-07-23T12:26:51.970 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.970 INFO:teuthology.orchestra.run.smithi133.stdout: mon allow pool delete = true 2020-07-23T12:26:51.971 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.971 INFO:teuthology.orchestra.run.smithi133.stdout: mon cluster log file level = debug 2020-07-23T12:26:51.971 INFO:teuthology.orchestra.run.smithi133.stdout: debug asserts on shutdown = true 2020-07-23T12:26:51.971 INFO:teuthology.orchestra.run.smithi133.stdout: mon host = 172.21.15.137 2020-07-23T12:26:51.971 INFO:teuthology.orchestra.run.smithi133.stdout: osd heartbeat grace = 40 2020-07-23T12:26:51.971 INFO:teuthology.orchestra.run.smithi133.stdout: fsid = 2f2a5b22-0949-421c-a9cb-4c87aa049a79 2020-07-23T12:26:51.972 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.972 INFO:teuthology.orchestra.run.smithi133.stdout:[osd] 2020-07-23T12:26:51.972 INFO:teuthology.orchestra.run.smithi133.stdout: osd journal size = 100 2020-07-23T12:26:51.972 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.972 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub load threshold = 5.0 2020-07-23T12:26:51.973 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub max interval = 600 2020-07-23T12:26:51.973 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.973 INFO:teuthology.orchestra.run.smithi133.stdout: osd recover clone overlap = true 2020-07-23T12:26:51.973 INFO:teuthology.orchestra.run.smithi133.stdout: osd recovery max chunk = 1048576 2020-07-23T12:26:51.973 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.974 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug shutdown = true 2020-07-23T12:26:51.974 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug op order = true 2020-07-23T12:26:51.974 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug verify stray on activate = true 2020-07-23T12:26:51.974 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.974 INFO:teuthology.orchestra.run.smithi133.stdout: osd open classes on start = true 2020-07-23T12:26:51.974 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug pg log writeout = true 2020-07-23T12:26:51.975 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.975 INFO:teuthology.orchestra.run.smithi133.stdout: osd deep scrub update digest min age = 30 2020-07-23T12:26:51.975 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.975 INFO:teuthology.orchestra.run.smithi133.stdout: osd map max advance = 10 2020-07-23T12:26:51.975 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.976 INFO:teuthology.orchestra.run.smithi133.stdout: journal zero on create = true 2020-07-23T12:26:51.976 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.976 INFO:teuthology.orchestra.run.smithi133.stdout: filestore ondisk finisher threads = 3 2020-07-23T12:26:51.976 INFO:teuthology.orchestra.run.smithi133.stdout: filestore apply finisher threads = 3 2020-07-23T12:26:51.976 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.976 INFO:teuthology.orchestra.run.smithi133.stdout: bdev debug aio = true 2020-07-23T12:26:51.977 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug misdirected ops = true 2020-07-23T12:26:51.977 INFO:teuthology.orchestra.run.smithi133.stdout: debug filestore = 20 2020-07-23T12:26:51.977 INFO:teuthology.orchestra.run.smithi133.stdout: debug journal = 20 2020-07-23T12:26:51.977 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 20 2020-07-23T12:26:51.977 INFO:teuthology.orchestra.run.smithi133.stdout: debug osd = 25 2020-07-23T12:26:51.978 INFO:teuthology.orchestra.run.smithi133.stdout: osd fast shutdown = False 2020-07-23T12:26:51.978 INFO:teuthology.orchestra.run.smithi133.stdout: rocksdb delete range threshold = 0 2020-07-23T12:26:51.978 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.978 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr] 2020-07-23T12:26:51.978 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2020-07-23T12:26:51.979 INFO:teuthology.orchestra.run.smithi133.stdout: debug mgr = 20 2020-07-23T12:26:51.979 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2020-07-23T12:26:51.979 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2020-07-23T12:26:51.979 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2020-07-23T12:26:51.979 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2020-07-23T12:26:51.979 INFO:teuthology.orchestra.run.smithi133.stdout: mgr/telemetry/nag = false 2020-07-23T12:26:51.980 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.980 INFO:teuthology.orchestra.run.smithi133.stdout:[mon] 2020-07-23T12:26:51.980 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2020-07-23T12:26:51.980 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2020-07-23T12:26:51.980 INFO:teuthology.orchestra.run.smithi133.stdout: debug paxos = 20 2020-07-23T12:26:51.981 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2020-07-23T12:26:51.981 INFO:teuthology.orchestra.run.smithi133.stdout: mon data avail warn = 5 2020-07-23T12:26:51.981 INFO:teuthology.orchestra.run.smithi133.stdout: mon mgr mkfs grace = 240 2020-07-23T12:26:51.981 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2020-07-23T12:26:51.981 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd reporter subtree level = osd 2020-07-23T12:26:51.982 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd prime pg temp = true 2020-07-23T12:26:51.982 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2020-07-23T12:26:51.982 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd crush smoke test = False 2020-07-23T12:26:51.982 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:26:51.982 INFO:teuthology.orchestra.run.smithi133.stdout:[client] 2020-07-23T12:26:51.982 INFO:teuthology.orchestra.run.smithi133.stdout: rgw cache enabled = true 2020-07-23T12:26:51.983 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable ops log = true 2020-07-23T12:26:51.983 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable usage log = true 2020-07-23T12:26:51.983 INFO:teuthology.orchestra.run.smithi133.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-07-23T12:26:51.983 INFO:teuthology.orchestra.run.smithi133.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-07-23T12:26:51.983 INFO:teuthology.orchestra.run.smithi133.stdout: debug rgw = 20 2020-07-23T12:26:51.984 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt require ssl = False 2020-07-23T12:26:51.984 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt s3 kms backend = testing 2020-07-23T12:26:51.984 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-23T12:26:51.984 INFO:teuthology.orchestra.run.smithi133.stdout: rgw curl low speed time = 300 2020-07-23T12:26:51.984 INFO:teuthology.orchestra.run.smithi133.stdout: rgw data log num shards = 4 2020-07-23T12:26:51.985 INFO:teuthology.orchestra.run.smithi133.stdout: rgw md log max shards = 4 2020-07-23T12:26:51.985 INFO:teuthology.orchestra.run.smithi133.stdout: rgw sync log trim interval = 0 2020-07-23T12:26:51.985 INFO:teuthology.orchestra.run.smithi133.stdout: setgroup = ceph 2020-07-23T12:26:51.985 INFO:teuthology.orchestra.run.smithi133.stdout: setuser = ceph 2020-07-23T12:26:51.985 INFO:teuthology.orchestra.run.smithi133.stdout:[mon.a] 2020-07-23T12:26:51.985 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.0] 2020-07-23T12:26:51.986 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 10 2020-07-23T12:26:51.986 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.1] 2020-07-23T12:26:51.986 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 1000 2020-07-23T12:26:51.986 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.2] 2020-07-23T12:26:51.986 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 10000 2020-07-23T12:26:51.994 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2020-07-23T12:26:51.994 INFO:teuthology.orchestra.run.smithi137.stdout: chdir = "" 2020-07-23T12:26:51.994 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-07-23T12:26:51.995 INFO:teuthology.orchestra.run.smithi137.stdout: auth supported = cephx 2020-07-23T12:26:51.995 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:51.995 INFO:teuthology.orchestra.run.smithi137.stdout: filestore xattr use omap = true 2020-07-23T12:26:51.995 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:51.995 INFO:teuthology.orchestra.run.smithi137.stdout: mon clock drift allowed = 1.000 2020-07-23T12:26:51.996 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:51.996 INFO:teuthology.orchestra.run.smithi137.stdout: osd crush chooseleaf type = 0 2020-07-23T12:26:51.996 INFO:teuthology.orchestra.run.smithi137.stdout: auth debug = true 2020-07-23T12:26:51.996 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:51.996 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on old message = true 2020-07-23T12:26:51.997 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on bug = true 2020-07-23T12:26:51.997 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:51.997 INFO:teuthology.orchestra.run.smithi137.stdout: mon max pg per osd = 10000 # >= luminous 2020-07-23T12:26:51.997 INFO:teuthology.orchestra.run.smithi137.stdout: mon pg warn max object skew = 0 2020-07-23T12:26:51.997 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:51.998 INFO:teuthology.orchestra.run.smithi137.stdout: # disable pg_autoscaler by default for new pools 2020-07-23T12:26:51.998 INFO:teuthology.orchestra.run.smithi137.stdout: osd_pool_default_pg_autoscale_mode = off 2020-07-23T12:26:51.998 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:51.998 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default size = 2 2020-07-23T12:26:51.998 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:51.999 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow primary affinity = true 2020-07-23T12:26:51.999 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow pg remap = true 2020-07-23T12:26:51.999 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on legacy crush tunables = false 2020-07-23T12:26:51.999 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on crush straw calc version zero = false 2020-07-23T12:26:51.999 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on no sortbitwise = false 2020-07-23T12:26:51.999 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on osd down out interval zero = false 2020-07-23T12:26:52.000 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on too few osds = false 2020-07-23T12:26:52.000 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-07-23T12:26:52.000 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_no_redundancy = false 2020-07-23T12:26:52.000 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:52.000 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-07-23T12:26:52.001 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:52.001 INFO:teuthology.orchestra.run.smithi137.stdout: osd default data pool replay window = 5 2020-07-23T12:26:52.001 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:52.001 INFO:teuthology.orchestra.run.smithi137.stdout: mon allow pool delete = true 2020-07-23T12:26:52.001 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:52.002 INFO:teuthology.orchestra.run.smithi137.stdout: mon cluster log file level = debug 2020-07-23T12:26:52.002 INFO:teuthology.orchestra.run.smithi137.stdout: debug asserts on shutdown = true 2020-07-23T12:26:52.002 INFO:teuthology.orchestra.run.smithi137.stdout: mon host = 172.21.15.137 2020-07-23T12:26:52.002 INFO:teuthology.orchestra.run.smithi137.stdout: osd heartbeat grace = 40 2020-07-23T12:26:52.002 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = 2f2a5b22-0949-421c-a9cb-4c87aa049a79 2020-07-23T12:26:52.002 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:52.003 INFO:teuthology.orchestra.run.smithi137.stdout:[osd] 2020-07-23T12:26:52.003 INFO:teuthology.orchestra.run.smithi137.stdout: osd journal size = 100 2020-07-23T12:26:52.003 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:52.003 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub load threshold = 5.0 2020-07-23T12:26:52.003 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub max interval = 600 2020-07-23T12:26:52.004 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:52.004 INFO:teuthology.orchestra.run.smithi137.stdout: osd recover clone overlap = true 2020-07-23T12:26:52.004 INFO:teuthology.orchestra.run.smithi137.stdout: osd recovery max chunk = 1048576 2020-07-23T12:26:52.004 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:52.004 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug shutdown = true 2020-07-23T12:26:52.005 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug op order = true 2020-07-23T12:26:52.005 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify stray on activate = true 2020-07-23T12:26:52.005 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:52.005 INFO:teuthology.orchestra.run.smithi137.stdout: osd open classes on start = true 2020-07-23T12:26:52.005 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug pg log writeout = true 2020-07-23T12:26:52.005 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:52.006 INFO:teuthology.orchestra.run.smithi137.stdout: osd deep scrub update digest min age = 30 2020-07-23T12:26:52.006 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:52.006 INFO:teuthology.orchestra.run.smithi137.stdout: osd map max advance = 10 2020-07-23T12:26:52.006 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:52.006 INFO:teuthology.orchestra.run.smithi137.stdout: journal zero on create = true 2020-07-23T12:26:52.007 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:52.007 INFO:teuthology.orchestra.run.smithi137.stdout: filestore ondisk finisher threads = 3 2020-07-23T12:26:52.007 INFO:teuthology.orchestra.run.smithi137.stdout: filestore apply finisher threads = 3 2020-07-23T12:26:52.007 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:52.007 INFO:teuthology.orchestra.run.smithi137.stdout: bdev debug aio = true 2020-07-23T12:26:52.008 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug misdirected ops = true 2020-07-23T12:26:52.008 INFO:teuthology.orchestra.run.smithi137.stdout: debug filestore = 20 2020-07-23T12:26:52.008 INFO:teuthology.orchestra.run.smithi137.stdout: debug journal = 20 2020-07-23T12:26:52.008 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 20 2020-07-23T12:26:52.008 INFO:teuthology.orchestra.run.smithi137.stdout: debug osd = 25 2020-07-23T12:26:52.008 INFO:teuthology.orchestra.run.smithi137.stdout: osd fast shutdown = False 2020-07-23T12:26:52.009 INFO:teuthology.orchestra.run.smithi137.stdout: rocksdb delete range threshold = 0 2020-07-23T12:26:52.009 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:52.009 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr] 2020-07-23T12:26:52.009 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2020-07-23T12:26:52.009 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgr = 20 2020-07-23T12:26:52.010 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2020-07-23T12:26:52.010 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2020-07-23T12:26:52.010 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2020-07-23T12:26:52.010 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2020-07-23T12:26:52.010 INFO:teuthology.orchestra.run.smithi137.stdout: mgr/telemetry/nag = false 2020-07-23T12:26:52.011 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:52.011 INFO:teuthology.orchestra.run.smithi137.stdout:[mon] 2020-07-23T12:26:52.011 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2020-07-23T12:26:52.011 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2020-07-23T12:26:52.011 INFO:teuthology.orchestra.run.smithi137.stdout: debug paxos = 20 2020-07-23T12:26:52.011 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2020-07-23T12:26:52.012 INFO:teuthology.orchestra.run.smithi137.stdout: mon data avail warn = 5 2020-07-23T12:26:52.012 INFO:teuthology.orchestra.run.smithi137.stdout: mon mgr mkfs grace = 240 2020-07-23T12:26:52.012 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2020-07-23T12:26:52.012 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd reporter subtree level = osd 2020-07-23T12:26:52.012 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd prime pg temp = true 2020-07-23T12:26:52.013 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2020-07-23T12:26:52.013 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd crush smoke test = False 2020-07-23T12:26:52.013 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:26:52.013 INFO:teuthology.orchestra.run.smithi137.stdout:[client] 2020-07-23T12:26:52.013 INFO:teuthology.orchestra.run.smithi137.stdout: rgw cache enabled = true 2020-07-23T12:26:52.014 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable ops log = true 2020-07-23T12:26:52.014 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable usage log = true 2020-07-23T12:26:52.014 INFO:teuthology.orchestra.run.smithi137.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-07-23T12:26:52.014 INFO:teuthology.orchestra.run.smithi137.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-07-23T12:26:52.014 INFO:teuthology.orchestra.run.smithi137.stdout: debug rgw = 20 2020-07-23T12:26:52.014 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt require ssl = False 2020-07-23T12:26:52.015 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt s3 kms backend = testing 2020-07-23T12:26:52.015 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-23T12:26:52.015 INFO:teuthology.orchestra.run.smithi137.stdout: rgw curl low speed time = 300 2020-07-23T12:26:52.015 INFO:teuthology.orchestra.run.smithi137.stdout: rgw data log num shards = 4 2020-07-23T12:26:52.016 INFO:teuthology.orchestra.run.smithi137.stdout: rgw md log max shards = 4 2020-07-23T12:26:52.016 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync log trim interval = 0 2020-07-23T12:26:52.016 INFO:teuthology.orchestra.run.smithi137.stdout: setgroup = ceph 2020-07-23T12:26:52.016 INFO:teuthology.orchestra.run.smithi137.stdout: setuser = ceph 2020-07-23T12:26:52.016 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.a] 2020-07-23T12:26:52.017 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.0] 2020-07-23T12:26:52.017 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 10 2020-07-23T12:26:52.017 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.1] 2020-07-23T12:26:52.017 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 1000 2020-07-23T12:26:52.017 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.2] 2020-07-23T12:26:52.018 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 10000 2020-07-23T12:26:52.019 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2020-07-23T12:26:52.019 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2020-07-23T12:26:52.106 INFO:tasks.ceph:Copying monmap to all nodes... 2020-07-23T12:26:52.115 DEBUG:teuthology.orchestra.remote:smithi137:/etc/ceph/c1.keyring is 206B 2020-07-23T12:26:52.132 DEBUG:teuthology.orchestra.remote:smithi137:/home/ubuntu/cephtest/c1.monmap is 179B 2020-07-23T12:26:52.142 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi137.front.sepia.ceph.com 2020-07-23T12:26:52.143 INFO:teuthology.orchestra.run.smithi137:> sudo sh -c 'cat > /etc/ceph/c1.keyring' && sudo chmod 0644 /etc/ceph/c1.keyring 2020-07-23T12:26:52.200 INFO:teuthology.orchestra.run.smithi137:> cat > /home/ubuntu/cephtest/c1.monmap 2020-07-23T12:26:52.289 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi133.front.sepia.ceph.com 2020-07-23T12:26:52.290 INFO:teuthology.orchestra.run.smithi133:> sudo sh -c 'cat > /etc/ceph/c1.keyring' && sudo chmod 0644 /etc/ceph/c1.keyring 2020-07-23T12:26:52.345 INFO:teuthology.orchestra.run.smithi133:> cat > /home/ubuntu/cephtest/c1.monmap 2020-07-23T12:26:52.431 INFO:tasks.ceph:Setting up mon nodes... 2020-07-23T12:26:52.431 INFO:tasks.ceph:Setting up mgr nodes... 2020-07-23T12:26:52.432 INFO:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2020-07-23T12:26:52.473 INFO:teuthology.orchestra.run.smithi137.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2020-07-23T12:26:52.475 INFO:tasks.ceph:Setting up mds nodes... 2020-07-23T12:26:52.475 INFO:tasks.ceph_client:Setting up client nodes... 2020-07-23T12:26:52.476 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2020-07-23T12:26:52.548 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/c1.client.0.keyring 2020-07-23T12:26:52.560 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2020-07-23T12:26:52.637 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/c1.client.1.keyring 2020-07-23T12:26:52.648 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-07-23T12:26:52.649 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-07-23T12:26:52.650 INFO:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2020-07-23T12:26:52.708 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-23T12:26:52.708 INFO:tasks.ceph:role: c1.osd.0 2020-07-23T12:26:52.709 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com 2020-07-23T12:26:52.709 INFO:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-07-23T12:26:53.249 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-07-23T12:26:53.249 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-23T12:26:53.249 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-23T12:26:53.249 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-23T12:26:53.250 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2020-07-23T12:26:53.250 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-23T12:26:53.250 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-23T12:26:53.250 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-23T12:26:53.250 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-23T12:26:53.251 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2020-07-23T12:26:53.252 INFO:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2020-07-23T12:26:53.341 INFO:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2020-07-23T12:26:53.353 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: /sbin/restorecon: command not found 2020-07-23T12:26:53.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-23T12:26:53.354 INFO:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2020-07-23T12:26:53.411 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-23T12:26:53.411 INFO:tasks.ceph:role: c1.osd.1 2020-07-23T12:26:53.412 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com 2020-07-23T12:26:53.412 INFO:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-07-23T12:26:53.938 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-07-23T12:26:53.938 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-23T12:26:53.939 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-23T12:26:53.939 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-23T12:26:53.939 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2020-07-23T12:26:53.939 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-23T12:26:53.939 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-23T12:26:53.940 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-23T12:26:53.940 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-23T12:26:53.941 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2020-07-23T12:26:53.941 INFO:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2020-07-23T12:26:53.979 INFO:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2020-07-23T12:26:54.029 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: /sbin/restorecon: command not found 2020-07-23T12:26:54.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-23T12:26:54.031 INFO:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2020-07-23T12:26:54.085 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-23T12:26:54.086 INFO:tasks.ceph:role: c1.osd.2 2020-07-23T12:26:54.086 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com 2020-07-23T12:26:54.087 INFO:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-07-23T12:26:54.606 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-07-23T12:26:54.606 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-23T12:26:54.607 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-23T12:26:54.607 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-23T12:26:54.607 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2020-07-23T12:26:54.607 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-23T12:26:54.607 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-23T12:26:54.608 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-23T12:26:54.608 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-23T12:26:54.609 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2020-07-23T12:26:54.609 INFO:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2020-07-23T12:26:54.648 INFO:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2020-07-23T12:26:54.706 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: /sbin/restorecon: command not found 2020-07-23T12:26:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-23T12:26:54.707 INFO:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2020-07-23T12:26:54.793 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:54.790+0000 7fc175d5ed80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2020-07-23T12:26:54.795 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:54.790+0000 7fc175d5ed80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2020-07-23T12:26:54.796 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:54.790+0000 7fc175d5ed80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-07-23T12:26:54.796 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:54.790+0000 7fc175d5ed80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-07-23T12:26:54.796 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:54.790+0000 7fc175d5ed80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-07-23T12:26:54.803 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:54.798+0000 7fc175d5ed80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2020-07-23T12:26:54.822 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:54.818+0000 7fc175d5ed80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-07-23T12:26:56.719 INFO:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2020-07-23T12:26:56.775 INFO:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2020-07-23T12:26:56.861 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:56.858+0000 7f46735d7d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2020-07-23T12:26:56.861 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:56.858+0000 7f46735d7d80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2020-07-23T12:26:56.862 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:56.858+0000 7f46735d7d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-07-23T12:26:56.862 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:56.858+0000 7f46735d7d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-07-23T12:26:56.862 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:56.858+0000 7f46735d7d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-07-23T12:26:56.863 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:56.858+0000 7f46735d7d80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2020-07-23T12:26:56.878 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:56.874+0000 7f46735d7d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-07-23T12:26:58.796 INFO:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2020-07-23T12:26:58.851 INFO:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2020-07-23T12:26:58.933 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:58.930+0000 7fe360542d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2020-07-23T12:26:58.934 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:58.930+0000 7fe360542d80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2020-07-23T12:26:58.934 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:58.930+0000 7fe360542d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-07-23T12:26:58.934 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:58.930+0000 7fe360542d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-07-23T12:26:58.935 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:58.930+0000 7fe360542d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-07-23T12:26:58.935 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:58.930+0000 7fe360542d80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2020-07-23T12:26:58.950 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:26:58.946+0000 7fe360542d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-07-23T12:27:00.851 INFO:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2020-07-23T12:27:00.908 INFO:tasks.ceph:Reading keys from all nodes... 2020-07-23T12:27:00.909 INFO:teuthology.orchestra.run.smithi137:> mktemp 2020-07-23T12:27:00.958 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.lZngT6SlHc 2020-07-23T12:27:00.959 INFO:teuthology.orchestra.run.smithi137:> sudo cp /var/lib/ceph/mgr/c1-x/keyring /tmp/tmp.lZngT6SlHc 2020-07-23T12:27:01.018 INFO:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.lZngT6SlHc 2020-07-23T12:27:01.123 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.lZngT6SlHc is 56B 2020-07-23T12:27:01.135 INFO:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.lZngT6SlHc 2020-07-23T12:27:01.141 INFO:teuthology.orchestra.run.smithi137:> mktemp 2020-07-23T12:27:01.190 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.uJRjDbX9om 2020-07-23T12:27:01.190 INFO:teuthology.orchestra.run.smithi137:> sudo cp /var/lib/ceph/osd/c1-0/keyring /tmp/tmp.uJRjDbX9om 2020-07-23T12:27:01.247 INFO:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.uJRjDbX9om 2020-07-23T12:27:01.351 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.uJRjDbX9om is 56B 2020-07-23T12:27:01.362 INFO:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.uJRjDbX9om 2020-07-23T12:27:01.369 INFO:teuthology.orchestra.run.smithi137:> mktemp 2020-07-23T12:27:01.419 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.ax7HnOOh7x 2020-07-23T12:27:01.420 INFO:teuthology.orchestra.run.smithi137:> sudo cp /var/lib/ceph/osd/c1-1/keyring /tmp/tmp.ax7HnOOh7x 2020-07-23T12:27:01.475 INFO:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.ax7HnOOh7x 2020-07-23T12:27:01.603 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.ax7HnOOh7x is 56B 2020-07-23T12:27:01.653 INFO:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.ax7HnOOh7x 2020-07-23T12:27:01.659 INFO:teuthology.orchestra.run.smithi137:> mktemp 2020-07-23T12:27:01.705 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.nEh3qkKs1N 2020-07-23T12:27:01.706 INFO:teuthology.orchestra.run.smithi137:> sudo cp /var/lib/ceph/osd/c1-2/keyring /tmp/tmp.nEh3qkKs1N 2020-07-23T12:27:01.763 INFO:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.nEh3qkKs1N 2020-07-23T12:27:01.872 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.nEh3qkKs1N is 56B 2020-07-23T12:27:01.927 INFO:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.nEh3qkKs1N 2020-07-23T12:27:01.985 DEBUG:teuthology.orchestra.remote:smithi137:/etc/ceph/c1.client.0.keyring is 59B 2020-07-23T12:27:02.007 DEBUG:teuthology.orchestra.remote:smithi137:/etc/ceph/c1.client.1.keyring is 59B 2020-07-23T12:27:02.017 INFO:tasks.ceph:Adding keys to all mons... 2020-07-23T12:27:02.018 INFO:teuthology.orchestra.run.smithi137:> sudo tee -a /etc/ceph/c1.keyring 2020-07-23T12:27:02.030 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr.x] 2020-07-23T12:27:02.030 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAMghlfHfdTHBAAzTBfs9NUnZCTY9hmIi0hPg== 2020-07-23T12:27:02.030 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.0] 2020-07-23T12:27:02.030 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAOghlf4FppLxAAfo81wZr2EVgMGqsctYLcaw== 2020-07-23T12:27:02.031 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.1] 2020-07-23T12:27:02.031 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAQghlf5NFuMxAAdcDRvQQ8uGfAHdmxRbvgaQ== 2020-07-23T12:27:02.031 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.2] 2020-07-23T12:27:02.031 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQASghlfCcS5NxAAYEF9yez38XwAWrE9tS/c0Q== 2020-07-23T12:27:02.031 INFO:teuthology.orchestra.run.smithi137.stdout:[client.0] 2020-07-23T12:27:02.031 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAMghlf+0TQIBAA11rOS/BpZy5hl/8764ja1g== 2020-07-23T12:27:02.032 INFO:teuthology.orchestra.run.smithi137.stdout:[client.1] 2020-07-23T12:27:02.032 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAMghlfCK0XJhAASwWO1VQeqeVgjnMXH2lDzA== 2020-07-23T12:27:02.033 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-07-23T12:27:02.120 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-23T12:27:02.205 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-23T12:27:02.289 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-23T12:27:02.378 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-07-23T12:27:02.478 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-07-23T12:27:02.526 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-07-23T12:27:02.528 INFO:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2020-07-23T12:27:02.582 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2020-07-23T12:27:02.686 INFO:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2020-07-23T12:27:02.700 INFO:teuthology.orchestra.run.smithi137:> rm -- /home/ubuntu/cephtest/c1.monmap 2020-07-23T12:27:02.750 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2020-07-23T12:27:02.751 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2020-07-23T12:27:02.752 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2020-07-23T12:27:02.798 INFO:tasks.ceph.c1.mon.a:Started 2020-07-23T12:27:02.798 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2020-07-23T12:27:02.799 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2020-07-23T12:27:02.799 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2020-07-23T12:27:02.801 INFO:tasks.ceph.c1.mgr.x:Started 2020-07-23T12:27:02.802 INFO:tasks.ceph:Setting crush tunables to default 2020-07-23T12:27:02.802 INFO:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c1 osd crush tunables default 2020-07-23T12:27:03.256 INFO:teuthology.orchestra.run.smithi137.stderr:adjusted tunables profile to default 2020-07-23T12:27:03.268 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2020-07-23T12:27:03.269 INFO:teuthology.orchestra.run.smithi137:> mktemp 2020-07-23T12:27:03.276 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.Nd30BWuv6i 2020-07-23T12:27:03.278 INFO:teuthology.orchestra.run.smithi137:> sudo cp /var/lib/ceph/osd/c1-0/fsid /tmp/tmp.Nd30BWuv6i 2020-07-23T12:27:03.335 INFO:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.Nd30BWuv6i 2020-07-23T12:27:03.440 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.Nd30BWuv6i is 37B 2020-07-23T12:27:03.495 INFO:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.Nd30BWuv6i 2020-07-23T12:27:03.502 INFO:teuthology.orchestra.run.smithi137:> mktemp 2020-07-23T12:27:03.550 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.IDTlnjOM1h 2020-07-23T12:27:03.551 INFO:teuthology.orchestra.run.smithi137:> sudo cp /var/lib/ceph/osd/c1-1/fsid /tmp/tmp.IDTlnjOM1h 2020-07-23T12:27:03.606 INFO:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.IDTlnjOM1h 2020-07-23T12:27:03.717 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.IDTlnjOM1h is 37B 2020-07-23T12:27:03.728 INFO:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.IDTlnjOM1h 2020-07-23T12:27:03.734 INFO:teuthology.orchestra.run.smithi137:> mktemp 2020-07-23T12:27:03.781 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.fyCwhb9RRu 2020-07-23T12:27:03.782 INFO:teuthology.orchestra.run.smithi137:> sudo cp /var/lib/ceph/osd/c1-2/fsid /tmp/tmp.fyCwhb9RRu 2020-07-23T12:27:03.840 INFO:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.fyCwhb9RRu 2020-07-23T12:27:03.951 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.fyCwhb9RRu is 37B 2020-07-23T12:27:03.961 INFO:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.fyCwhb9RRu 2020-07-23T12:27:03.966 INFO:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c1 osd new 79ac5d8d-b106-4412-9ba8-9f3f5d65605c 0 2020-07-23T12:27:04.208 INFO:teuthology.orchestra.run.smithi137.stdout:0 2020-07-23T12:27:04.220 INFO:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c1 osd new 11116a77-0f16-4040-a17d-5fab256b9fc7 1 2020-07-23T12:27:04.414 INFO:teuthology.orchestra.run.smithi137.stdout:1 2020-07-23T12:27:04.425 INFO:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c1 osd new 07eeea6c-f85f-455d-a5cb-6c4f657d7ce7 2 2020-07-23T12:27:04.626 INFO:teuthology.orchestra.run.smithi137.stdout:2 2020-07-23T12:27:04.637 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2020-07-23T12:27:04.637 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2020-07-23T12:27:04.643 INFO:tasks.ceph.c1.osd.0:Started 2020-07-23T12:27:04.643 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2020-07-23T12:27:04.643 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2020-07-23T12:27:04.645 INFO:tasks.ceph.c1.osd.1:Started 2020-07-23T12:27:04.646 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2020-07-23T12:27:04.646 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2020-07-23T12:27:04.648 INFO:tasks.ceph.c1.osd.2:Started 2020-07-23T12:27:04.648 INFO:tasks.ceph:Waiting for OSDs to come up 2020-07-23T12:27:04.952 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-07-23T12:27:05.129 INFO:teuthology.misc.health.smithi137.stdout: 2020-07-23T12:27:05.129 INFO:teuthology.misc.health.smithi137.stdout:{"epoch":5,"fsid":"2f2a5b22-0949-421c-a9cb-4c87aa049a79","created":"2020-07-23T12:27:02.919772+0000","modified":"2020-07-23T12:27:04.626023+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"79ac5d8d-b106-4412-9ba8-9f3f5d65605c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"11116a77-0f16-4040-a17d-5fab256b9fc7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"07eeea6c-f85f-455d-a5cb-6c4f657d7ce7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-23T12:27:05.138 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-07-23T12:27:05.259 INFO:tasks.ceph.c1.osd.1.smithi137.stderr:2020-07-23T12:27:05.254+0000 7f36af7f5d80 -1 Falling back to public interface 2020-07-23T12:27:05.260 INFO:tasks.ceph.c1.osd.2.smithi137.stderr:2020-07-23T12:27:05.254+0000 7f3614eedd80 -1 Falling back to public interface 2020-07-23T12:27:05.277 INFO:tasks.ceph.c1.osd.0.smithi137.stderr:2020-07-23T12:27:05.274+0000 7f79838e3d80 -1 Falling back to public interface 2020-07-23T12:27:05.648 INFO:tasks.ceph.c1.osd.2.smithi137.stderr:2020-07-23T12:27:05.642+0000 7f3614eedd80 -1 osd.2 0 log_to_monitors {default=true} 2020-07-23T12:27:05.881 INFO:tasks.ceph.c1.osd.1.smithi137.stderr:2020-07-23T12:27:05.878+0000 7f36af7f5d80 -1 osd.1 0 log_to_monitors {default=true} 2020-07-23T12:27:05.901 INFO:tasks.ceph.c1.osd.0.smithi137.stderr:2020-07-23T12:27:05.898+0000 7f79838e3d80 -1 osd.0 0 log_to_monitors {default=true} 2020-07-23T12:27:06.961 INFO:tasks.ceph.c1.osd.0.smithi137.stderr:2020-07-23T12:27:06.954+0000 7f796d330700 -1 osd.0 0 waiting for initial osdmap 2020-07-23T12:27:06.962 INFO:tasks.ceph.c1.osd.1.smithi137.stderr:2020-07-23T12:27:06.954+0000 7f3699242700 -1 osd.1 0 waiting for initial osdmap 2020-07-23T12:27:06.962 INFO:tasks.ceph.c1.osd.2.smithi137.stderr:2020-07-23T12:27:06.954+0000 7f35fe93a700 -1 osd.2 0 waiting for initial osdmap 2020-07-23T12:27:11.442 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-07-23T12:27:11.706 INFO:teuthology.misc.health.smithi137.stdout: 2020-07-23T12:27:11.707 INFO:teuthology.misc.health.smithi137.stdout:{"epoch":10,"fsid":"2f2a5b22-0949-421c-a9cb-4c87aa049a79","created":"2020-07-23T12:27:02.919772+0000","modified":"2020-07-23T12:27:10.583693+0000","last_up_change":"2020-07-23T12:27:07.959600+0000","last_in_change":"2020-07-23T12:27:07.959600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-23T12:27:06.593197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"79ac5d8d-b106-4412-9ba8-9f3f5d65605c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":11785},{"type":"v1","addr":"172.21.15.137:6817","nonce":11785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":11785},{"type":"v1","addr":"172.21.15.137:6819","nonce":11785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":11785},{"type":"v1","addr":"172.21.15.137:6823","nonce":11785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":11785},{"type":"v1","addr":"172.21.15.137:6821","nonce":11785}]},"public_addr":"172.21.15.137:6817/11785","cluster_addr":"172.21.15.137:6819/11785","heartbeat_back_addr":"172.21.15.137:6823/11785","heartbeat_front_addr":"172.21.15.137:6821/11785","state":["exists","up"]},{"osd":1,"uuid":"11116a77-0f16-4040-a17d-5fab256b9fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":11787},{"type":"v1","addr":"172.21.15.137:6801","nonce":11787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6803","nonce":11787},{"type":"v1","addr":"172.21.15.137:6804","nonce":11787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":11787},{"type":"v1","addr":"172.21.15.137:6811","nonce":11787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":11787},{"type":"v1","addr":"172.21.15.137:6808","nonce":11787}]},"public_addr":"172.21.15.137:6801/11787","cluster_addr":"172.21.15.137:6804/11787","heartbeat_back_addr":"172.21.15.137:6811/11787","heartbeat_front_addr":"172.21.15.137:6808/11787","state":["exists","up"]},{"osd":2,"uuid":"07eeea6c-f85f-455d-a5cb-6c4f657d7ce7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":11786},{"type":"v1","addr":"172.21.15.137:6805","nonce":11786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6807","nonce":11786},{"type":"v1","addr":"172.21.15.137:6809","nonce":11786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":11786},{"type":"v1","addr":"172.21.15.137:6815","nonce":11786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":11786},{"type":"v1","addr":"172.21.15.137:6813","nonce":11786}]},"public_addr":"172.21.15.137:6805/11786","cluster_addr":"172.21.15.137:6809/11786","heartbeat_back_addr":"172.21.15.137:6815/11786","heartbeat_front_addr":"172.21.15.137:6813/11786","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:06.919428+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:06.916152+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:06.685448+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-23T12:27:11.718 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-07-23T12:27:11.719 INFO:tasks.ceph:Creating RBD pool 2020-07-23T12:27:11.719 INFO:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c1 osd pool create rbd 8 2020-07-23T12:27:12.248 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'rbd' created 2020-07-23T12:27:12.262 INFO:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-07-23T12:27:13.263 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rbd' on pool 'rbd' 2020-07-23T12:27:13.277 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2020-07-23T12:27:13.278 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-07-23T12:27:13.279 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-07-23T12:27:13.538 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:13.539 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":12,"fsid":"2f2a5b22-0949-421c-a9cb-4c87aa049a79","created":"2020-07-23T12:27:02.919772+0000","modified":"2020-07-23T12:27:13.240560+0000","last_up_change":"2020-07-23T12:27:07.959600+0000","last_in_change":"2020-07-23T12:27:07.959600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-23T12:27:06.593197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-23T12:27:11.981765+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"79ac5d8d-b106-4412-9ba8-9f3f5d65605c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":11785},{"type":"v1","addr":"172.21.15.137:6817","nonce":11785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":11785},{"type":"v1","addr":"172.21.15.137:6819","nonce":11785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":11785},{"type":"v1","addr":"172.21.15.137:6823","nonce":11785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":11785},{"type":"v1","addr":"172.21.15.137:6821","nonce":11785}]},"public_addr":"172.21.15.137:6817/11785","cluster_addr":"172.21.15.137:6819/11785","heartbeat_back_addr":"172.21.15.137:6823/11785","heartbeat_front_addr":"172.21.15.137:6821/11785","state":["exists","up"]},{"osd":1,"uuid":"11116a77-0f16-4040-a17d-5fab256b9fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":11787},{"type":"v1","addr":"172.21.15.137:6801","nonce":11787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6803","nonce":11787},{"type":"v1","addr":"172.21.15.137:6804","nonce":11787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":11787},{"type":"v1","addr":"172.21.15.137:6811","nonce":11787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":11787},{"type":"v1","addr":"172.21.15.137:6808","nonce":11787}]},"public_addr":"172.21.15.137:6801/11787","cluster_addr":"172.21.15.137:6804/11787","heartbeat_back_addr":"172.21.15.137:6811/11787","heartbeat_front_addr":"172.21.15.137:6808/11787","state":["exists","up"]},{"osd":2,"uuid":"07eeea6c-f85f-455d-a5cb-6c4f657d7ce7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":11786},{"type":"v1","addr":"172.21.15.137:6805","nonce":11786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6807","nonce":11786},{"type":"v1","addr":"172.21.15.137:6809","nonce":11786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":11786},{"type":"v1","addr":"172.21.15.137:6815","nonce":11786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":11786},{"type":"v1","addr":"172.21.15.137:6813","nonce":11786}]},"public_addr":"172.21.15.137:6805/11786","cluster_addr":"172.21.15.137:6809/11786","heartbeat_back_addr":"172.21.15.137:6815/11786","heartbeat_front_addr":"172.21.15.137:6813/11786","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:06.919428+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:06.916152+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:06.685448+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-23T12:27:13.551 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-07-23T12:27:06.593197+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-07-23T12:27:11.981765+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-07-23T12:27:13.552 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get device_health_metrics pg_num 2020-07-23T12:27:13.819 INFO:teuthology.orchestra.run.smithi137.stdout:pg_num: 1 2020-07-23T12:27:13.833 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get rbd pg_num 2020-07-23T12:27:14.104 INFO:teuthology.orchestra.run.smithi137.stdout:pg_num: 8 2020-07-23T12:27:14.124 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2020-07-23T12:27:14.125 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2020-07-23T12:27:14.125 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early mgr dump --format=json 2020-07-23T12:27:14.421 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:14.433 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":11576},{"type":"v1","addr":"172.21.15.137:6825","nonce":11576}]},"active_addr":"172.21.15.137:6825/11576","active_change":"2020-07-23T12:27:06.564520+0000","active_mgr_features":4540138292836696063,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":2834068709}]},{"addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":3605598106}]},{"addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":2848146991}]}]}} 2020-07-23T12:27:14.437 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2020-07-23T12:27:14.437 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2020-07-23T12:27:14.438 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-07-23T12:27:14.704 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:14.704 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":13,"fsid":"2f2a5b22-0949-421c-a9cb-4c87aa049a79","created":"2020-07-23T12:27:02.919772+0000","modified":"2020-07-23T12:27:14.249429+0000","last_up_change":"2020-07-23T12:27:07.959600+0000","last_in_change":"2020-07-23T12:27:07.959600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-23T12:27:06.593197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-23T12:27:11.981765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"79ac5d8d-b106-4412-9ba8-9f3f5d65605c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":11785},{"type":"v1","addr":"172.21.15.137:6817","nonce":11785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":11785},{"type":"v1","addr":"172.21.15.137:6819","nonce":11785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":11785},{"type":"v1","addr":"172.21.15.137:6823","nonce":11785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":11785},{"type":"v1","addr":"172.21.15.137:6821","nonce":11785}]},"public_addr":"172.21.15.137:6817/11785","cluster_addr":"172.21.15.137:6819/11785","heartbeat_back_addr":"172.21.15.137:6823/11785","heartbeat_front_addr":"172.21.15.137:6821/11785","state":["exists","up"]},{"osd":1,"uuid":"11116a77-0f16-4040-a17d-5fab256b9fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":11787},{"type":"v1","addr":"172.21.15.137:6801","nonce":11787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6803","nonce":11787},{"type":"v1","addr":"172.21.15.137:6804","nonce":11787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":11787},{"type":"v1","addr":"172.21.15.137:6811","nonce":11787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":11787},{"type":"v1","addr":"172.21.15.137:6808","nonce":11787}]},"public_addr":"172.21.15.137:6801/11787","cluster_addr":"172.21.15.137:6804/11787","heartbeat_back_addr":"172.21.15.137:6811/11787","heartbeat_front_addr":"172.21.15.137:6808/11787","state":["exists","up"]},{"osd":2,"uuid":"07eeea6c-f85f-455d-a5cb-6c4f657d7ce7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":11786},{"type":"v1","addr":"172.21.15.137:6805","nonce":11786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6807","nonce":11786},{"type":"v1","addr":"172.21.15.137:6809","nonce":11786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":11786},{"type":"v1","addr":"172.21.15.137:6815","nonce":11786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":11786},{"type":"v1","addr":"172.21.15.137:6813","nonce":11786}]},"public_addr":"172.21.15.137:6805/11786","cluster_addr":"172.21.15.137:6809/11786","heartbeat_back_addr":"172.21.15.137:6815/11786","heartbeat_front_addr":"172.21.15.137:6813/11786","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:06.919428+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:06.916152+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:06.685448+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-23T12:27:14.717 INFO:tasks.ceph.ceph_manager.c1:all up! 2020-07-23T12:27:14.718 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-07-23T12:27:14.977 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:14.977 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":13,"fsid":"2f2a5b22-0949-421c-a9cb-4c87aa049a79","created":"2020-07-23T12:27:02.919772+0000","modified":"2020-07-23T12:27:14.249429+0000","last_up_change":"2020-07-23T12:27:07.959600+0000","last_in_change":"2020-07-23T12:27:07.959600+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-23T12:27:06.593197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-23T12:27:11.981765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"79ac5d8d-b106-4412-9ba8-9f3f5d65605c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":11785},{"type":"v1","addr":"172.21.15.137:6817","nonce":11785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":11785},{"type":"v1","addr":"172.21.15.137:6819","nonce":11785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":11785},{"type":"v1","addr":"172.21.15.137:6823","nonce":11785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":11785},{"type":"v1","addr":"172.21.15.137:6821","nonce":11785}]},"public_addr":"172.21.15.137:6817/11785","cluster_addr":"172.21.15.137:6819/11785","heartbeat_back_addr":"172.21.15.137:6823/11785","heartbeat_front_addr":"172.21.15.137:6821/11785","state":["exists","up"]},{"osd":1,"uuid":"11116a77-0f16-4040-a17d-5fab256b9fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":11787},{"type":"v1","addr":"172.21.15.137:6801","nonce":11787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6803","nonce":11787},{"type":"v1","addr":"172.21.15.137:6804","nonce":11787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":11787},{"type":"v1","addr":"172.21.15.137:6811","nonce":11787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":11787},{"type":"v1","addr":"172.21.15.137:6808","nonce":11787}]},"public_addr":"172.21.15.137:6801/11787","cluster_addr":"172.21.15.137:6804/11787","heartbeat_back_addr":"172.21.15.137:6811/11787","heartbeat_front_addr":"172.21.15.137:6808/11787","state":["exists","up"]},{"osd":2,"uuid":"07eeea6c-f85f-455d-a5cb-6c4f657d7ce7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":11786},{"type":"v1","addr":"172.21.15.137:6805","nonce":11786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6807","nonce":11786},{"type":"v1","addr":"172.21.15.137:6809","nonce":11786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":11786},{"type":"v1","addr":"172.21.15.137:6815","nonce":11786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":11786},{"type":"v1","addr":"172.21.15.137:6813","nonce":11786}]},"public_addr":"172.21.15.137:6805/11786","cluster_addr":"172.21.15.137:6809/11786","heartbeat_back_addr":"172.21.15.137:6815/11786","heartbeat_front_addr":"172.21.15.137:6813/11786","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:06.919428+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:06.916152+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:06.685448+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-23T12:27:14.991 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.0 flush_pg_stats 2020-07-23T12:27:15.141 INFO:teuthology.orchestra.run.smithi137.stdout:34359738371 2020-07-23T12:27:15.141 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.1 flush_pg_stats 2020-07-23T12:27:15.295 INFO:teuthology.orchestra.run.smithi137.stdout:34359738371 2020-07-23T12:27:15.296 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.2 flush_pg_stats 2020-07-23T12:27:15.450 INFO:teuthology.orchestra.run.smithi137.stdout:34359738372 2020-07-23T12:27:15.451 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-07-23T12:27:15.727 INFO:teuthology.orchestra.run.smithi137.stdout:34359738370 2020-07-23T12:27:15.740 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2020-07-23T12:27:16.741 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-07-23T12:27:17.018 INFO:teuthology.orchestra.run.smithi137.stdout:34359738370 2020-07-23T12:27:17.031 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2020-07-23T12:27:18.033 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-07-23T12:27:18.299 INFO:teuthology.orchestra.run.smithi137.stdout:34359738371 2020-07-23T12:27:18.313 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.0 2020-07-23T12:27:18.313 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.1 2020-07-23T12:27:18.579 INFO:teuthology.orchestra.run.smithi137.stdout:34359738371 2020-07-23T12:27:18.592 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.1 2020-07-23T12:27:18.593 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.2 2020-07-23T12:27:18.859 INFO:teuthology.orchestra.run.smithi137.stdout:34359738372 2020-07-23T12:27:18.878 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.2 2020-07-23T12:27:18.879 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2020-07-23T12:27:18.880 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-07-23T12:27:19.139 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:19.142 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2020-07-23T12:27:19.152 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-07-23T12:27:18.575367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.324070"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.274091+0000","last_change":"2020-07-23T12:27:13.272673+0000","last_active":"2020-07-23T12:27:13.274091+0000","last_peered":"2020-07-23T12:27:13.274091+0000","last_clean":"2020-07-23T12:27:13.274091+0000","last_became_active":"2020-07-23T12:27:13.272278+0000","last_became_peered":"2020-07-23T12:27:13.272278+0000","last_unstale":"2020-07-23T12:27:13.274091+0000","last_undegraded":"2020-07-23T12:27:13.274091+0000","last_fullsized":"2020-07-23T12:27:13.274091+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_clean_scrub_stamp":"2020-07-23T12:27:12.234849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.273068+0000","last_change":"2020-07-23T12:27:13.272456+0000","last_active":"2020-07-23T12:27:13.273068+0000","last_peered":"2020-07-23T12:27:13.273068+0000","last_clean":"2020-07-23T12:27:13.273068+0000","last_became_active":"2020-07-23T12:27:13.272046+0000","last_became_peered":"2020-07-23T12:27:13.272046+0000","last_unstale":"2020-07-23T12:27:13.273068+0000","last_undegraded":"2020-07-23T12:27:13.273068+0000","last_fullsized":"2020-07-23T12:27:13.273068+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_clean_scrub_stamp":"2020-07-23T12:27:12.234849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.271962+0000","last_change":"2020-07-23T12:27:13.267448+0000","last_active":"2020-07-23T12:27:13.271962+0000","last_peered":"2020-07-23T12:27:13.271962+0000","last_clean":"2020-07-23T12:27:13.271962+0000","last_became_active":"2020-07-23T12:27:13.267027+0000","last_became_peered":"2020-07-23T12:27:13.267027+0000","last_unstale":"2020-07-23T12:27:13.271962+0000","last_undegraded":"2020-07-23T12:27:13.271962+0000","last_fullsized":"2020-07-23T12:27:13.271962+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_clean_scrub_stamp":"2020-07-23T12:27:12.234849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.271460+0000","last_change":"2020-07-23T12:27:13.267287+0000","last_active":"2020-07-23T12:27:13.271460+0000","last_peered":"2020-07-23T12:27:13.271460+0000","last_clean":"2020-07-23T12:27:13.271460+0000","last_became_active":"2020-07-23T12:27:13.266849+0000","last_became_peered":"2020-07-23T12:27:13.266849+0000","last_unstale":"2020-07-23T12:27:13.271460+0000","last_undegraded":"2020-07-23T12:27:13.271460+0000","last_fullsized":"2020-07-23T12:27:13.271460+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_clean_scrub_stamp":"2020-07-23T12:27:12.234849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.261738+0000","last_change":"2020-07-23T12:27:09.948823+0000","last_active":"2020-07-23T12:27:13.261738+0000","last_peered":"2020-07-23T12:27:13.261738+0000","last_clean":"2020-07-23T12:27:13.261738+0000","last_became_active":"2020-07-23T12:27:09.947069+0000","last_became_peered":"2020-07-23T12:27:09.947069+0000","last_unstale":"2020-07-23T12:27:13.261738+0000","last_undegraded":"2020-07-23T12:27:13.261738+0000","last_fullsized":"2020-07-23T12:27:13.261738+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:06.944994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:06.944994+0000","last_clean_scrub_stamp":"2020-07-23T12:27:06.944994+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.271985+0000","last_change":"2020-07-23T12:27:13.269928+0000","last_active":"2020-07-23T12:27:13.271985+0000","last_peered":"2020-07-23T12:27:13.271985+0000","last_clean":"2020-07-23T12:27:13.271985+0000","last_became_active":"2020-07-23T12:27:13.269513+0000","last_became_peered":"2020-07-23T12:27:13.269513+0000","last_unstale":"2020-07-23T12:27:13.271985+0000","last_undegraded":"2020-07-23T12:27:13.271985+0000","last_fullsized":"2020-07-23T12:27:13.271985+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_clean_scrub_stamp":"2020-07-23T12:27:12.234849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.271442+0000","last_change":"2020-07-23T12:27:13.265049+0000","last_active":"2020-07-23T12:27:13.271442+0000","last_peered":"2020-07-23T12:27:13.271442+0000","last_clean":"2020-07-23T12:27:13.271442+0000","last_became_active":"2020-07-23T12:27:13.264695+0000","last_became_peered":"2020-07-23T12:27:13.264695+0000","last_unstale":"2020-07-23T12:27:13.271442+0000","last_undegraded":"2020-07-23T12:27:13.271442+0000","last_fullsized":"2020-07-23T12:27:13.271442+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_clean_scrub_stamp":"2020-07-23T12:27:12.234849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.271762+0000","last_change":"2020-07-23T12:27:13.267218+0000","last_active":"2020-07-23T12:27:13.271762+0000","last_peered":"2020-07-23T12:27:13.271762+0000","last_clean":"2020-07-23T12:27:13.271762+0000","last_became_active":"2020-07-23T12:27:13.266865+0000","last_became_peered":"2020-07-23T12:27:13.266865+0000","last_unstale":"2020-07-23T12:27:13.271762+0000","last_undegraded":"2020-07-23T12:27:13.271762+0000","last_fullsized":"2020-07-23T12:27:13.271762+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_clean_scrub_stamp":"2020-07-23T12:27:12.234849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.276676+0000","last_change":"2020-07-23T12:27:13.272960+0000","last_active":"2020-07-23T12:27:13.276676+0000","last_peered":"2020-07-23T12:27:13.276676+0000","last_clean":"2020-07-23T12:27:13.276676+0000","last_became_active":"2020-07-23T12:27:13.272490+0000","last_became_peered":"2020-07-23T12:27:13.272490+0000","last_unstale":"2020-07-23T12:27:13.276676+0000","last_undegraded":"2020-07-23T12:27:13.276676+0000","last_fullsized":"2020-07-23T12:27:13.276676+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_clean_scrub_stamp":"2020-07-23T12:27:12.234849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.746}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.752}]}]}]}} 2020-07-23T12:27:19.153 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-07-23T12:27:19.409 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:19.411 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2020-07-23T12:27:19.421 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-07-23T12:27:18.575367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.324070"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.274091+0000","last_change":"2020-07-23T12:27:13.272673+0000","last_active":"2020-07-23T12:27:13.274091+0000","last_peered":"2020-07-23T12:27:13.274091+0000","last_clean":"2020-07-23T12:27:13.274091+0000","last_became_active":"2020-07-23T12:27:13.272278+0000","last_became_peered":"2020-07-23T12:27:13.272278+0000","last_unstale":"2020-07-23T12:27:13.274091+0000","last_undegraded":"2020-07-23T12:27:13.274091+0000","last_fullsized":"2020-07-23T12:27:13.274091+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_clean_scrub_stamp":"2020-07-23T12:27:12.234849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.273068+0000","last_change":"2020-07-23T12:27:13.272456+0000","last_active":"2020-07-23T12:27:13.273068+0000","last_peered":"2020-07-23T12:27:13.273068+0000","last_clean":"2020-07-23T12:27:13.273068+0000","last_became_active":"2020-07-23T12:27:13.272046+0000","last_became_peered":"2020-07-23T12:27:13.272046+0000","last_unstale":"2020-07-23T12:27:13.273068+0000","last_undegraded":"2020-07-23T12:27:13.273068+0000","last_fullsized":"2020-07-23T12:27:13.273068+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_clean_scrub_stamp":"2020-07-23T12:27:12.234849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.271962+0000","last_change":"2020-07-23T12:27:13.267448+0000","last_active":"2020-07-23T12:27:13.271962+0000","last_peered":"2020-07-23T12:27:13.271962+0000","last_clean":"2020-07-23T12:27:13.271962+0000","last_became_active":"2020-07-23T12:27:13.267027+0000","last_became_peered":"2020-07-23T12:27:13.267027+0000","last_unstale":"2020-07-23T12:27:13.271962+0000","last_undegraded":"2020-07-23T12:27:13.271962+0000","last_fullsized":"2020-07-23T12:27:13.271962+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_clean_scrub_stamp":"2020-07-23T12:27:12.234849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.271460+0000","last_change":"2020-07-23T12:27:13.267287+0000","last_active":"2020-07-23T12:27:13.271460+0000","last_peered":"2020-07-23T12:27:13.271460+0000","last_clean":"2020-07-23T12:27:13.271460+0000","last_became_active":"2020-07-23T12:27:13.266849+0000","last_became_peered":"2020-07-23T12:27:13.266849+0000","last_unstale":"2020-07-23T12:27:13.271460+0000","last_undegraded":"2020-07-23T12:27:13.271460+0000","last_fullsized":"2020-07-23T12:27:13.271460+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_clean_scrub_stamp":"2020-07-23T12:27:12.234849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.261738+0000","last_change":"2020-07-23T12:27:09.948823+0000","last_active":"2020-07-23T12:27:13.261738+0000","last_peered":"2020-07-23T12:27:13.261738+0000","last_clean":"2020-07-23T12:27:13.261738+0000","last_became_active":"2020-07-23T12:27:09.947069+0000","last_became_peered":"2020-07-23T12:27:09.947069+0000","last_unstale":"2020-07-23T12:27:13.261738+0000","last_undegraded":"2020-07-23T12:27:13.261738+0000","last_fullsized":"2020-07-23T12:27:13.261738+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:06.944994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:06.944994+0000","last_clean_scrub_stamp":"2020-07-23T12:27:06.944994+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.271985+0000","last_change":"2020-07-23T12:27:13.269928+0000","last_active":"2020-07-23T12:27:13.271985+0000","last_peered":"2020-07-23T12:27:13.271985+0000","last_clean":"2020-07-23T12:27:13.271985+0000","last_became_active":"2020-07-23T12:27:13.269513+0000","last_became_peered":"2020-07-23T12:27:13.269513+0000","last_unstale":"2020-07-23T12:27:13.271985+0000","last_undegraded":"2020-07-23T12:27:13.271985+0000","last_fullsized":"2020-07-23T12:27:13.271985+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_clean_scrub_stamp":"2020-07-23T12:27:12.234849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.271442+0000","last_change":"2020-07-23T12:27:13.265049+0000","last_active":"2020-07-23T12:27:13.271442+0000","last_peered":"2020-07-23T12:27:13.271442+0000","last_clean":"2020-07-23T12:27:13.271442+0000","last_became_active":"2020-07-23T12:27:13.264695+0000","last_became_peered":"2020-07-23T12:27:13.264695+0000","last_unstale":"2020-07-23T12:27:13.271442+0000","last_undegraded":"2020-07-23T12:27:13.271442+0000","last_fullsized":"2020-07-23T12:27:13.271442+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_clean_scrub_stamp":"2020-07-23T12:27:12.234849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.271762+0000","last_change":"2020-07-23T12:27:13.267218+0000","last_active":"2020-07-23T12:27:13.271762+0000","last_peered":"2020-07-23T12:27:13.271762+0000","last_clean":"2020-07-23T12:27:13.271762+0000","last_became_active":"2020-07-23T12:27:13.266865+0000","last_became_peered":"2020-07-23T12:27:13.266865+0000","last_unstale":"2020-07-23T12:27:13.271762+0000","last_undegraded":"2020-07-23T12:27:13.271762+0000","last_fullsized":"2020-07-23T12:27:13.271762+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_clean_scrub_stamp":"2020-07-23T12:27:12.234849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:13.276676+0000","last_change":"2020-07-23T12:27:13.272960+0000","last_active":"2020-07-23T12:27:13.276676+0000","last_peered":"2020-07-23T12:27:13.276676+0000","last_clean":"2020-07-23T12:27:13.276676+0000","last_became_active":"2020-07-23T12:27:13.272490+0000","last_became_peered":"2020-07-23T12:27:13.272490+0000","last_unstale":"2020-07-23T12:27:13.276676+0000","last_undegraded":"2020-07-23T12:27:13.276676+0000","last_fullsized":"2020-07-23T12:27:13.276676+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:12.234849+0000","last_clean_scrub_stamp":"2020-07-23T12:27:12.234849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.746}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.752}]}]}]}} 2020-07-23T12:27:19.423 INFO:tasks.ceph.ceph_manager.c1:clean! 2020-07-23T12:27:19.423 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2020-07-23T12:27:19.423 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2020-07-23T12:27:19.424 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early health --format=json 2020-07-23T12:27:19.732 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:19.745 INFO:teuthology.orchestra.run.smithi137.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-07-23T12:27:19.746 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2020-07-23T12:27:19.746 INFO:teuthology.run_tasks:Running task ceph... 2020-07-23T12:27:19.758 INFO:tasks.ceph:Creating ceph cluster c2... 2020-07-23T12:27:19.759 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-07-23T12:27:19.759 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2020-07-20_05:05:02-rgw-octopus-distro-basic-smithi/5242029', 'branch': 'octopus', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '5242029', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2020-07-20_05:05:02-rgw-octopus-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'octopus'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': 'bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'octopus', 'sha1': 'bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 1000, 'repo': 'git://git.ceph.com/ceph.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': 'bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'octopus', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_octopus/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph.git', 'suite_sha1': 'bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'bdc0fb0c5eb43ffa20f86c88093d654732d1e1ac', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.20760', 'targets': {'smithi137.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDujhwHQQYYd0i9/NuavIxWtoPBK11AcchJKBZRpQXB49v1LqcluhHxhXvo8P17ads2mEYs5fiU6zWmywcGE9JJ2skLImeTRfEaTwuAmAMKfGZ1Lv9jX3EHAGViMTYcpkY59w462r7A+d5gt9JY/8aNlWdNdQgEY3GA7h6iZuixavroXd28UB3wzTVLOrCCTpYDMhDlvmioE63/iXbtwUE0UYhEAsnpWk7nx8kLw+nRrHs1QPlH447Vpqpb/zx0zMeJ9e/Xvu3KyHImTa8P9eV59CW6j3FnufcbWAl/NxI6lgjXfqJQO0f4PZSlM95kS94Yp90nqD2FzaxXrQpXhJvH', 'smithi133.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDTunE7kpOkFCjYvZc7sF3Pj+bcPT2jzsHSBbPaEA/tW/bjDZRdd8emTC4L0fxv0CG02maN33JHAxJdBoLaUZuGuGaExQ0q3xlxBxslCwAn7h+uWJdYS3ZTtHDriAcNG1tK0zAy7hkF1WTJptRrQGhEMp4R+cKZqkNnrl71yMvrTCxK+RVdWdx92U4i476ggc0dQZdKst52B9oUJ6dU4fu2Ivn5GjWiYk1Voy/TyRaG8TgVEyvpKJ38UURB1Axb/P/60/Xn9WDsWp3k25z5uw3dzdEC6z3z8i8jUpHssvD0l7CGxxRZiwkt3MoKH4XY5J7cI1Ynm9QjI2YcQqqh6P8z'}} 2020-07-23T12:27:19.760 INFO:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-07-23T12:27:19.763 INFO:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-07-23T12:27:19.770 INFO:teuthology.orchestra.run.smithi133:> sudo install -d -m0777 -- /var/run/ceph 2020-07-23T12:27:19.810 INFO:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/run/ceph 2020-07-23T12:27:19.876 DEBUG:teuthology.orchestra.remote:smithi133:/scratch_devs is 72B 2020-07-23T12:27:19.890 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-07-23T12:27:19.891 INFO:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-07-23T12:27:19.898 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-07-23T12:27:19.899 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-23T12:27:19.899 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-07-23T12:27:19.899 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-23T12:27:19.899 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2020-07-23 12:24:22.494641379 +0000 2020-07-23T12:27:19.899 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2020-07-23 12:24:22.366705362 +0000 2020-07-23T12:27:19.900 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2020-07-23 12:24:22.366705362 +0000 2020-07-23T12:27:19.900 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2020-07-23T12:27:19.907 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2020-07-23T12:27:19.907 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2020-07-23T12:27:19.907 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000384048 s, 1.3 MB/s 2020-07-23T12:27:19.912 INFO:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-07-23T12:27:19.959 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-07-23T12:27:19.960 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-23T12:27:19.960 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 6h/6d Inode: 348 Links: 1 2020-07-23T12:27:19.960 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-23T12:27:19.960 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2020-07-23 12:24:22.734521411 +0000 2020-07-23T12:27:19.960 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2020-07-23 12:24:22.606585394 +0000 2020-07-23T12:27:19.961 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2020-07-23 12:24:22.606585394 +0000 2020-07-23T12:27:19.961 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2020-07-23T12:27:19.970 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2020-07-23T12:27:19.970 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2020-07-23T12:27:19.970 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000362287 s, 1.4 MB/s 2020-07-23T12:27:19.973 INFO:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-07-23T12:27:20.023 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-07-23T12:27:20.024 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-23T12:27:20.024 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-07-23T12:27:20.024 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-23T12:27:20.024 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2020-07-23 12:24:22.854461427 +0000 2020-07-23T12:27:20.024 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2020-07-23 12:24:22.850463427 +0000 2020-07-23T12:27:20.025 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2020-07-23 12:24:22.850463427 +0000 2020-07-23T12:27:20.025 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2020-07-23T12:27:20.032 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2020-07-23T12:27:20.032 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2020-07-23T12:27:20.032 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000384941 s, 1.3 MB/s 2020-07-23T12:27:20.036 INFO:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-07-23T12:27:20.083 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-07-23T12:27:20.084 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-07-23T12:27:20.084 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-07-23T12:27:20.084 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-07-23T12:27:20.084 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2020-07-23 12:24:23.130323464 +0000 2020-07-23T12:27:20.084 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2020-07-23 12:24:23.126325464 +0000 2020-07-23T12:27:20.085 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2020-07-23 12:24:23.126325464 +0000 2020-07-23T12:27:20.085 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: - 2020-07-23T12:27:20.093 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2020-07-23T12:27:20.093 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2020-07-23T12:27:20.093 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000402417 s, 1.3 MB/s 2020-07-23T12:27:20.097 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-23T12:27:20.098 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi133.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-07-23T12:27:20.098 INFO:tasks.ceph:Generating config... 2020-07-23T12:27:20.100 INFO:tasks.ceph:[client] debug rgw = 20 2020-07-23T12:27:20.100 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-07-23T12:27:20.100 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-07-23T12:27:20.100 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-23T12:27:20.101 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-07-23T12:27:20.101 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-07-23T12:27:20.101 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-07-23T12:27:20.102 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-07-23T12:27:20.102 INFO:tasks.ceph:[client] setgroup = ceph 2020-07-23T12:27:20.102 INFO:tasks.ceph:[client] setuser = ceph 2020-07-23T12:27:20.103 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-07-23T12:27:20.103 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-07-23T12:27:20.103 INFO:tasks.ceph:[mgr] debug ms = 1 2020-07-23T12:27:20.103 INFO:tasks.ceph:[mon] debug mon = 20 2020-07-23T12:27:20.104 INFO:tasks.ceph:[mon] debug ms = 1 2020-07-23T12:27:20.104 INFO:tasks.ceph:[mon] debug paxos = 20 2020-07-23T12:27:20.104 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-07-23T12:27:20.105 INFO:tasks.ceph:[osd] debug filestore = 20 2020-07-23T12:27:20.105 INFO:tasks.ceph:[osd] debug journal = 20 2020-07-23T12:27:20.105 INFO:tasks.ceph:[osd] debug ms = 20 2020-07-23T12:27:20.105 INFO:tasks.ceph:[osd] debug osd = 25 2020-07-23T12:27:20.106 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-07-23T12:27:20.106 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-07-23T12:27:20.106 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-07-23T12:27:20.107 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-07-23T12:27:20.107 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-07-23T12:27:20.107 INFO:tasks.ceph:Setting up c2.mon.a... 2020-07-23T12:27:20.108 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2020-07-23T12:27:20.176 INFO:teuthology.orchestra.run.smithi133.stdout:creating /etc/ceph/c2.keyring 2020-07-23T12:27:20.178 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2020-07-23T12:27:20.261 INFO:teuthology.orchestra.run.smithi133:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-07-23T12:27:20.316 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.133')] 2020-07-23T12:27:20.316 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.133 --print /home/ubuntu/cephtest/c2.monmap 2020-07-23T12:27:20.384 INFO:teuthology.orchestra.run.smithi133.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2020-07-23T12:27:20.384 INFO:teuthology.orchestra.run.smithi133.stdout:monmaptool: generated fsid 2fb80b8c-a980-4374-ad78-15e596842194 2020-07-23T12:27:20.384 INFO:teuthology.orchestra.run.smithi133.stdout:epoch 0 2020-07-23T12:27:20.384 INFO:teuthology.orchestra.run.smithi133.stdout:fsid 2fb80b8c-a980-4374-ad78-15e596842194 2020-07-23T12:27:20.385 INFO:teuthology.orchestra.run.smithi133.stdout:last_changed 2020-07-23T12:27:20.387064+0000 2020-07-23T12:27:20.385 INFO:teuthology.orchestra.run.smithi133.stdout:created 2020-07-23T12:27:20.387064+0000 2020-07-23T12:27:20.385 INFO:teuthology.orchestra.run.smithi133.stdout:min_mon_release 0 (unknown) 2020-07-23T12:27:20.385 INFO:teuthology.orchestra.run.smithi133.stdout:0: [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon.a 2020-07-23T12:27:20.385 INFO:teuthology.orchestra.run.smithi133.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2020-07-23T12:27:20.387 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 2fb80b8c-a980-4374-ad78-15e596842194... 2020-07-23T12:27:20.388 INFO:teuthology.orchestra.run.smithi133:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-07-23T12:27:20.430 INFO:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-07-23T12:27:20.460 INFO:teuthology.orchestra.run.smithi133.stdout:[global] 2020-07-23T12:27:20.460 INFO:teuthology.orchestra.run.smithi133.stdout: chdir = "" 2020-07-23T12:27:20.460 INFO:teuthology.orchestra.run.smithi133.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-07-23T12:27:20.461 INFO:teuthology.orchestra.run.smithi133.stdout: auth supported = cephx 2020-07-23T12:27:20.461 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.461 INFO:teuthology.orchestra.run.smithi133.stdout: filestore xattr use omap = true 2020-07-23T12:27:20.461 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.461 INFO:teuthology.orchestra.run.smithi133.stdout: mon clock drift allowed = 1.000 2020-07-23T12:27:20.462 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.462 INFO:teuthology.orchestra.run.smithi133.stdout: osd crush chooseleaf type = 0 2020-07-23T12:27:20.462 INFO:teuthology.orchestra.run.smithi133.stdout: auth debug = true 2020-07-23T12:27:20.462 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.462 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on old message = true 2020-07-23T12:27:20.463 INFO:teuthology.orchestra.run.smithi133.stdout: ms die on bug = true 2020-07-23T12:27:20.463 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.463 INFO:teuthology.orchestra.run.smithi133.stdout: mon max pg per osd = 10000 # >= luminous 2020-07-23T12:27:20.463 INFO:teuthology.orchestra.run.smithi133.stdout: mon pg warn max object skew = 0 2020-07-23T12:27:20.463 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.464 INFO:teuthology.orchestra.run.smithi133.stdout: # disable pg_autoscaler by default for new pools 2020-07-23T12:27:20.464 INFO:teuthology.orchestra.run.smithi133.stdout: osd_pool_default_pg_autoscale_mode = off 2020-07-23T12:27:20.464 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.464 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default size = 2 2020-07-23T12:27:20.464 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.464 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow primary affinity = true 2020-07-23T12:27:20.465 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd allow pg remap = true 2020-07-23T12:27:20.465 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on legacy crush tunables = false 2020-07-23T12:27:20.465 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on crush straw calc version zero = false 2020-07-23T12:27:20.465 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on no sortbitwise = false 2020-07-23T12:27:20.465 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on osd down out interval zero = false 2020-07-23T12:27:20.466 INFO:teuthology.orchestra.run.smithi133.stdout: mon warn on too few osds = false 2020-07-23T12:27:20.466 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-07-23T12:27:20.466 INFO:teuthology.orchestra.run.smithi133.stdout: mon_warn_on_pool_no_redundancy = false 2020-07-23T12:27:20.466 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.466 INFO:teuthology.orchestra.run.smithi133.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-07-23T12:27:20.467 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.467 INFO:teuthology.orchestra.run.smithi133.stdout: osd default data pool replay window = 5 2020-07-23T12:27:20.467 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.467 INFO:teuthology.orchestra.run.smithi133.stdout: mon allow pool delete = true 2020-07-23T12:27:20.467 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.467 INFO:teuthology.orchestra.run.smithi133.stdout: mon cluster log file level = debug 2020-07-23T12:27:20.468 INFO:teuthology.orchestra.run.smithi133.stdout: debug asserts on shutdown = true 2020-07-23T12:27:20.468 INFO:teuthology.orchestra.run.smithi133.stdout: mon host = 172.21.15.133 2020-07-23T12:27:20.468 INFO:teuthology.orchestra.run.smithi133.stdout: osd heartbeat grace = 40 2020-07-23T12:27:20.468 INFO:teuthology.orchestra.run.smithi133.stdout: fsid = 2fb80b8c-a980-4374-ad78-15e596842194 2020-07-23T12:27:20.468 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.469 INFO:teuthology.orchestra.run.smithi133.stdout:[osd] 2020-07-23T12:27:20.469 INFO:teuthology.orchestra.run.smithi133.stdout: osd journal size = 100 2020-07-23T12:27:20.469 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.469 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub load threshold = 5.0 2020-07-23T12:27:20.469 INFO:teuthology.orchestra.run.smithi133.stdout: osd scrub max interval = 600 2020-07-23T12:27:20.470 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.470 INFO:teuthology.orchestra.run.smithi133.stdout: osd recover clone overlap = true 2020-07-23T12:27:20.470 INFO:teuthology.orchestra.run.smithi133.stdout: osd recovery max chunk = 1048576 2020-07-23T12:27:20.470 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.470 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug shutdown = true 2020-07-23T12:27:20.470 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug op order = true 2020-07-23T12:27:20.471 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug verify stray on activate = true 2020-07-23T12:27:20.471 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.471 INFO:teuthology.orchestra.run.smithi133.stdout: osd open classes on start = true 2020-07-23T12:27:20.471 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug pg log writeout = true 2020-07-23T12:27:20.471 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.472 INFO:teuthology.orchestra.run.smithi133.stdout: osd deep scrub update digest min age = 30 2020-07-23T12:27:20.472 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.472 INFO:teuthology.orchestra.run.smithi133.stdout: osd map max advance = 10 2020-07-23T12:27:20.472 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.472 INFO:teuthology.orchestra.run.smithi133.stdout: journal zero on create = true 2020-07-23T12:27:20.473 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.473 INFO:teuthology.orchestra.run.smithi133.stdout: filestore ondisk finisher threads = 3 2020-07-23T12:27:20.473 INFO:teuthology.orchestra.run.smithi133.stdout: filestore apply finisher threads = 3 2020-07-23T12:27:20.473 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.473 INFO:teuthology.orchestra.run.smithi133.stdout: bdev debug aio = true 2020-07-23T12:27:20.476 INFO:teuthology.orchestra.run.smithi133.stdout: osd debug misdirected ops = true 2020-07-23T12:27:20.476 INFO:teuthology.orchestra.run.smithi133.stdout: debug filestore = 20 2020-07-23T12:27:20.476 INFO:teuthology.orchestra.run.smithi133.stdout: debug journal = 20 2020-07-23T12:27:20.476 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 20 2020-07-23T12:27:20.476 INFO:teuthology.orchestra.run.smithi133.stdout: debug osd = 25 2020-07-23T12:27:20.477 INFO:teuthology.orchestra.run.smithi133.stdout: osd fast shutdown = False 2020-07-23T12:27:20.477 INFO:teuthology.orchestra.run.smithi133.stdout: rocksdb delete range threshold = 0 2020-07-23T12:27:20.477 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.477 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr] 2020-07-23T12:27:20.477 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2020-07-23T12:27:20.478 INFO:teuthology.orchestra.run.smithi133.stdout: debug mgr = 20 2020-07-23T12:27:20.478 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2020-07-23T12:27:20.478 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2020-07-23T12:27:20.478 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2020-07-23T12:27:20.478 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2020-07-23T12:27:20.479 INFO:teuthology.orchestra.run.smithi133.stdout: mgr/telemetry/nag = false 2020-07-23T12:27:20.479 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.479 INFO:teuthology.orchestra.run.smithi133.stdout:[mon] 2020-07-23T12:27:20.479 INFO:teuthology.orchestra.run.smithi133.stdout: debug ms = 1 2020-07-23T12:27:20.479 INFO:teuthology.orchestra.run.smithi133.stdout: debug mon = 20 2020-07-23T12:27:20.480 INFO:teuthology.orchestra.run.smithi133.stdout: debug paxos = 20 2020-07-23T12:27:20.480 INFO:teuthology.orchestra.run.smithi133.stdout: debug auth = 20 2020-07-23T12:27:20.480 INFO:teuthology.orchestra.run.smithi133.stdout: mon data avail warn = 5 2020-07-23T12:27:20.480 INFO:teuthology.orchestra.run.smithi133.stdout: mon mgr mkfs grace = 240 2020-07-23T12:27:20.480 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min pgs per osd = 4 2020-07-23T12:27:20.480 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd reporter subtree level = osd 2020-07-23T12:27:20.481 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd prime pg temp = true 2020-07-23T12:27:20.481 INFO:teuthology.orchestra.run.smithi133.stdout: mon reweight min bytes per osd = 10 2020-07-23T12:27:20.481 INFO:teuthology.orchestra.run.smithi133.stdout: mon osd crush smoke test = False 2020-07-23T12:27:20.481 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:20.481 INFO:teuthology.orchestra.run.smithi133.stdout:[client] 2020-07-23T12:27:20.482 INFO:teuthology.orchestra.run.smithi133.stdout: rgw cache enabled = true 2020-07-23T12:27:20.482 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable ops log = true 2020-07-23T12:27:20.482 INFO:teuthology.orchestra.run.smithi133.stdout: rgw enable usage log = true 2020-07-23T12:27:20.482 INFO:teuthology.orchestra.run.smithi133.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-07-23T12:27:20.482 INFO:teuthology.orchestra.run.smithi133.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-07-23T12:27:20.483 INFO:teuthology.orchestra.run.smithi133.stdout: debug rgw = 20 2020-07-23T12:27:20.483 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt require ssl = False 2020-07-23T12:27:20.483 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt s3 kms backend = testing 2020-07-23T12:27:20.483 INFO:teuthology.orchestra.run.smithi133.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-23T12:27:20.483 INFO:teuthology.orchestra.run.smithi133.stdout: rgw curl low speed time = 300 2020-07-23T12:27:20.483 INFO:teuthology.orchestra.run.smithi133.stdout: rgw data log num shards = 4 2020-07-23T12:27:20.484 INFO:teuthology.orchestra.run.smithi133.stdout: rgw md log max shards = 4 2020-07-23T12:27:20.484 INFO:teuthology.orchestra.run.smithi133.stdout: rgw sync log trim interval = 0 2020-07-23T12:27:20.484 INFO:teuthology.orchestra.run.smithi133.stdout: setgroup = ceph 2020-07-23T12:27:20.484 INFO:teuthology.orchestra.run.smithi133.stdout: setuser = ceph 2020-07-23T12:27:20.484 INFO:teuthology.orchestra.run.smithi133.stdout:[mon.a] 2020-07-23T12:27:20.485 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.0] 2020-07-23T12:27:20.485 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 10 2020-07-23T12:27:20.485 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.1] 2020-07-23T12:27:20.485 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 1000 2020-07-23T12:27:20.485 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.2] 2020-07-23T12:27:20.486 INFO:teuthology.orchestra.run.smithi133.stdout: osd_max_omap_entries_per_request = 10000 2020-07-23T12:27:20.487 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2020-07-23T12:27:20.487 INFO:teuthology.orchestra.run.smithi137.stdout: chdir = "" 2020-07-23T12:27:20.488 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-07-23T12:27:20.488 INFO:teuthology.orchestra.run.smithi137.stdout: auth supported = cephx 2020-07-23T12:27:20.488 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.488 INFO:teuthology.orchestra.run.smithi137.stdout: filestore xattr use omap = true 2020-07-23T12:27:20.488 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.489 INFO:teuthology.orchestra.run.smithi137.stdout: mon clock drift allowed = 1.000 2020-07-23T12:27:20.489 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.489 INFO:teuthology.orchestra.run.smithi137.stdout: osd crush chooseleaf type = 0 2020-07-23T12:27:20.489 INFO:teuthology.orchestra.run.smithi137.stdout: auth debug = true 2020-07-23T12:27:20.489 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.490 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on old message = true 2020-07-23T12:27:20.490 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on bug = true 2020-07-23T12:27:20.490 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.490 INFO:teuthology.orchestra.run.smithi137.stdout: mon max pg per osd = 10000 # >= luminous 2020-07-23T12:27:20.490 INFO:teuthology.orchestra.run.smithi137.stdout: mon pg warn max object skew = 0 2020-07-23T12:27:20.491 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.491 INFO:teuthology.orchestra.run.smithi137.stdout: # disable pg_autoscaler by default for new pools 2020-07-23T12:27:20.491 INFO:teuthology.orchestra.run.smithi137.stdout: osd_pool_default_pg_autoscale_mode = off 2020-07-23T12:27:20.491 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.491 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default size = 2 2020-07-23T12:27:20.492 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.492 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow primary affinity = true 2020-07-23T12:27:20.492 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow pg remap = true 2020-07-23T12:27:20.493 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on legacy crush tunables = false 2020-07-23T12:27:20.493 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on crush straw calc version zero = false 2020-07-23T12:27:20.493 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on no sortbitwise = false 2020-07-23T12:27:20.493 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on osd down out interval zero = false 2020-07-23T12:27:20.493 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on too few osds = false 2020-07-23T12:27:20.493 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-07-23T12:27:20.494 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_no_redundancy = false 2020-07-23T12:27:20.494 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.494 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-07-23T12:27:20.494 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.494 INFO:teuthology.orchestra.run.smithi137.stdout: osd default data pool replay window = 5 2020-07-23T12:27:20.495 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.495 INFO:teuthology.orchestra.run.smithi137.stdout: mon allow pool delete = true 2020-07-23T12:27:20.495 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.495 INFO:teuthology.orchestra.run.smithi137.stdout: mon cluster log file level = debug 2020-07-23T12:27:20.495 INFO:teuthology.orchestra.run.smithi137.stdout: debug asserts on shutdown = true 2020-07-23T12:27:20.496 INFO:teuthology.orchestra.run.smithi137.stdout: mon host = 172.21.15.133 2020-07-23T12:27:20.496 INFO:teuthology.orchestra.run.smithi137.stdout: osd heartbeat grace = 40 2020-07-23T12:27:20.496 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = 2fb80b8c-a980-4374-ad78-15e596842194 2020-07-23T12:27:20.496 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.496 INFO:teuthology.orchestra.run.smithi137.stdout:[osd] 2020-07-23T12:27:20.496 INFO:teuthology.orchestra.run.smithi137.stdout: osd journal size = 100 2020-07-23T12:27:20.497 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.497 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub load threshold = 5.0 2020-07-23T12:27:20.497 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub max interval = 600 2020-07-23T12:27:20.497 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.497 INFO:teuthology.orchestra.run.smithi137.stdout: osd recover clone overlap = true 2020-07-23T12:27:20.498 INFO:teuthology.orchestra.run.smithi137.stdout: osd recovery max chunk = 1048576 2020-07-23T12:27:20.498 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.498 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug shutdown = true 2020-07-23T12:27:20.498 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug op order = true 2020-07-23T12:27:20.498 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify stray on activate = true 2020-07-23T12:27:20.498 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.499 INFO:teuthology.orchestra.run.smithi137.stdout: osd open classes on start = true 2020-07-23T12:27:20.499 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug pg log writeout = true 2020-07-23T12:27:20.499 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.499 INFO:teuthology.orchestra.run.smithi137.stdout: osd deep scrub update digest min age = 30 2020-07-23T12:27:20.499 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.500 INFO:teuthology.orchestra.run.smithi137.stdout: osd map max advance = 10 2020-07-23T12:27:20.500 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.500 INFO:teuthology.orchestra.run.smithi137.stdout: journal zero on create = true 2020-07-23T12:27:20.500 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.500 INFO:teuthology.orchestra.run.smithi137.stdout: filestore ondisk finisher threads = 3 2020-07-23T12:27:20.500 INFO:teuthology.orchestra.run.smithi137.stdout: filestore apply finisher threads = 3 2020-07-23T12:27:20.501 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.501 INFO:teuthology.orchestra.run.smithi137.stdout: bdev debug aio = true 2020-07-23T12:27:20.501 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug misdirected ops = true 2020-07-23T12:27:20.501 INFO:teuthology.orchestra.run.smithi137.stdout: debug filestore = 20 2020-07-23T12:27:20.501 INFO:teuthology.orchestra.run.smithi137.stdout: debug journal = 20 2020-07-23T12:27:20.502 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 20 2020-07-23T12:27:20.502 INFO:teuthology.orchestra.run.smithi137.stdout: debug osd = 25 2020-07-23T12:27:20.502 INFO:teuthology.orchestra.run.smithi137.stdout: osd fast shutdown = False 2020-07-23T12:27:20.502 INFO:teuthology.orchestra.run.smithi137.stdout: rocksdb delete range threshold = 0 2020-07-23T12:27:20.502 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.503 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr] 2020-07-23T12:27:20.503 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2020-07-23T12:27:20.503 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgr = 20 2020-07-23T12:27:20.503 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2020-07-23T12:27:20.503 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2020-07-23T12:27:20.503 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2020-07-23T12:27:20.504 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2020-07-23T12:27:20.504 INFO:teuthology.orchestra.run.smithi137.stdout: mgr/telemetry/nag = false 2020-07-23T12:27:20.504 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.504 INFO:teuthology.orchestra.run.smithi137.stdout:[mon] 2020-07-23T12:27:20.504 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2020-07-23T12:27:20.505 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2020-07-23T12:27:20.505 INFO:teuthology.orchestra.run.smithi137.stdout: debug paxos = 20 2020-07-23T12:27:20.505 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2020-07-23T12:27:20.505 INFO:teuthology.orchestra.run.smithi137.stdout: mon data avail warn = 5 2020-07-23T12:27:20.505 INFO:teuthology.orchestra.run.smithi137.stdout: mon mgr mkfs grace = 240 2020-07-23T12:27:20.506 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2020-07-23T12:27:20.507 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd reporter subtree level = osd 2020-07-23T12:27:20.507 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd prime pg temp = true 2020-07-23T12:27:20.507 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2020-07-23T12:27:20.507 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd crush smoke test = False 2020-07-23T12:27:20.507 INFO:teuthology.orchestra.run.smithi137.stdout: 2020-07-23T12:27:20.508 INFO:teuthology.orchestra.run.smithi137.stdout:[client] 2020-07-23T12:27:20.508 INFO:teuthology.orchestra.run.smithi137.stdout: rgw cache enabled = true 2020-07-23T12:27:20.508 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable ops log = true 2020-07-23T12:27:20.508 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable usage log = true 2020-07-23T12:27:20.508 INFO:teuthology.orchestra.run.smithi137.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-07-23T12:27:20.509 INFO:teuthology.orchestra.run.smithi137.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-07-23T12:27:20.509 INFO:teuthology.orchestra.run.smithi137.stdout: debug rgw = 20 2020-07-23T12:27:20.509 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt require ssl = False 2020-07-23T12:27:20.509 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt s3 kms backend = testing 2020-07-23T12:27:20.509 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-07-23T12:27:20.509 INFO:teuthology.orchestra.run.smithi137.stdout: rgw curl low speed time = 300 2020-07-23T12:27:20.510 INFO:teuthology.orchestra.run.smithi137.stdout: rgw data log num shards = 4 2020-07-23T12:27:20.510 INFO:teuthology.orchestra.run.smithi137.stdout: rgw md log max shards = 4 2020-07-23T12:27:20.510 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync log trim interval = 0 2020-07-23T12:27:20.510 INFO:teuthology.orchestra.run.smithi137.stdout: setgroup = ceph 2020-07-23T12:27:20.510 INFO:teuthology.orchestra.run.smithi137.stdout: setuser = ceph 2020-07-23T12:27:20.511 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.a] 2020-07-23T12:27:20.511 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.0] 2020-07-23T12:27:20.511 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 10 2020-07-23T12:27:20.511 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.1] 2020-07-23T12:27:20.511 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 1000 2020-07-23T12:27:20.511 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.2] 2020-07-23T12:27:20.512 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 10000 2020-07-23T12:27:20.512 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2020-07-23T12:27:20.513 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2020-07-23T12:27:20.574 INFO:tasks.ceph:Copying monmap to all nodes... 2020-07-23T12:27:20.580 DEBUG:teuthology.orchestra.remote:smithi133:/etc/ceph/c2.keyring is 206B 2020-07-23T12:27:20.597 DEBUG:teuthology.orchestra.remote:smithi133:/home/ubuntu/cephtest/c2.monmap is 179B 2020-07-23T12:27:20.607 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi137.front.sepia.ceph.com 2020-07-23T12:27:20.607 INFO:teuthology.orchestra.run.smithi137:> sudo sh -c 'cat > /etc/ceph/c2.keyring' && sudo chmod 0644 /etc/ceph/c2.keyring 2020-07-23T12:27:20.664 INFO:teuthology.orchestra.run.smithi137:> cat > /home/ubuntu/cephtest/c2.monmap 2020-07-23T12:27:20.757 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi133.front.sepia.ceph.com 2020-07-23T12:27:20.758 INFO:teuthology.orchestra.run.smithi133:> sudo sh -c 'cat > /etc/ceph/c2.keyring' && sudo chmod 0644 /etc/ceph/c2.keyring 2020-07-23T12:27:20.817 INFO:teuthology.orchestra.run.smithi133:> cat > /home/ubuntu/cephtest/c2.monmap 2020-07-23T12:27:20.906 INFO:tasks.ceph:Setting up mon nodes... 2020-07-23T12:27:20.907 INFO:tasks.ceph:Setting up mgr nodes... 2020-07-23T12:27:20.907 INFO:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2020-07-23T12:27:20.992 INFO:teuthology.orchestra.run.smithi133.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2020-07-23T12:27:20.994 INFO:tasks.ceph:Setting up mds nodes... 2020-07-23T12:27:20.994 INFO:tasks.ceph_client:Setting up client nodes... 2020-07-23T12:27:20.995 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2020-07-23T12:27:21.071 INFO:teuthology.orchestra.run.smithi133.stdout:creating /etc/ceph/c2.client.0.keyring 2020-07-23T12:27:21.082 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2020-07-23T12:27:21.157 INFO:teuthology.orchestra.run.smithi133.stdout:creating /etc/ceph/c2.client.1.keyring 2020-07-23T12:27:21.169 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-07-23T12:27:21.170 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi133.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-07-23T12:27:21.170 INFO:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2020-07-23T12:27:21.228 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-23T12:27:21.229 INFO:tasks.ceph:role: c2.osd.0 2020-07-23T12:27:21.229 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi133.front.sepia.ceph.com 2020-07-23T12:27:21.230 INFO:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-07-23T12:27:21.747 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-07-23T12:27:21.748 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-23T12:27:21.748 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-23T12:27:21.749 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-23T12:27:21.749 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2020-07-23T12:27:21.749 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-23T12:27:21.749 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-23T12:27:21.750 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-23T12:27:21.751 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-23T12:27:21.752 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2020-07-23T12:27:21.753 INFO:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2020-07-23T12:27:21.834 INFO:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2020-07-23T12:27:21.846 INFO:teuthology.orchestra.run.smithi133.stderr:sudo: /sbin/restorecon: command not found 2020-07-23T12:27:21.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-23T12:27:21.847 INFO:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2020-07-23T12:27:21.915 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-23T12:27:21.915 INFO:tasks.ceph:role: c2.osd.1 2020-07-23T12:27:21.915 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi133.front.sepia.ceph.com 2020-07-23T12:27:21.916 INFO:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-07-23T12:27:22.452 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-07-23T12:27:22.453 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-23T12:27:22.453 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-23T12:27:22.453 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-23T12:27:22.453 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2020-07-23T12:27:22.453 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-23T12:27:22.454 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-23T12:27:22.454 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-23T12:27:22.454 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-23T12:27:22.455 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2020-07-23T12:27:22.456 INFO:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2020-07-23T12:27:22.490 INFO:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2020-07-23T12:27:22.544 INFO:teuthology.orchestra.run.smithi133.stderr:sudo: /sbin/restorecon: command not found 2020-07-23T12:27:22.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-23T12:27:22.546 INFO:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2020-07-23T12:27:22.600 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-07-23T12:27:22.600 INFO:tasks.ceph:role: c2.osd.2 2020-07-23T12:27:22.600 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi133.front.sepia.ceph.com 2020-07-23T12:27:22.601 INFO:teuthology.orchestra.run.smithi133:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-07-23T12:27:23.138 INFO:teuthology.orchestra.run.smithi133.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-07-23T12:27:23.138 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 attr=2, projid32bit=1 2020-07-23T12:27:23.138 INFO:teuthology.orchestra.run.smithi133.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-07-23T12:27:23.139 INFO:teuthology.orchestra.run.smithi133.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-07-23T12:27:23.139 INFO:teuthology.orchestra.run.smithi133.stdout: = sunit=0 swidth=0 blks 2020-07-23T12:27:23.139 INFO:teuthology.orchestra.run.smithi133.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-07-23T12:27:23.139 INFO:teuthology.orchestra.run.smithi133.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-07-23T12:27:23.139 INFO:teuthology.orchestra.run.smithi133.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-07-23T12:27:23.140 INFO:teuthology.orchestra.run.smithi133.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-07-23T12:27:23.140 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi133.front.sepia.ceph.com -o noatime 2020-07-23T12:27:23.140 INFO:teuthology.orchestra.run.smithi133:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2020-07-23T12:27:23.193 INFO:teuthology.orchestra.run.smithi133:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2020-07-23T12:27:23.249 INFO:teuthology.orchestra.run.smithi133.stderr:sudo: /sbin/restorecon: command not found 2020-07-23T12:27:23.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-07-23T12:27:23.251 INFO:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2020-07-23T12:27:23.337 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:23.336+0000 7f000c236d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2020-07-23T12:27:23.338 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:23.336+0000 7f000c236d80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2020-07-23T12:27:23.338 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:23.336+0000 7f000c236d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-07-23T12:27:23.338 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:23.336+0000 7f000c236d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-07-23T12:27:23.339 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:23.336+0000 7f000c236d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-07-23T12:27:23.342 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:23.340+0000 7f000c236d80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2020-07-23T12:27:23.360 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:23.360+0000 7f000c236d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-07-23T12:27:25.244 INFO:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2020-07-23T12:27:25.300 INFO:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2020-07-23T12:27:25.391 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:25.384+0000 7f6a5904dd80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2020-07-23T12:27:25.392 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:25.384+0000 7f6a5904dd80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2020-07-23T12:27:25.392 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:25.384+0000 7f6a5904dd80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-07-23T12:27:25.392 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:25.384+0000 7f6a5904dd80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-07-23T12:27:25.392 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:25.384+0000 7f6a5904dd80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-07-23T12:27:25.393 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:25.384+0000 7f6a5904dd80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2020-07-23T12:27:25.403 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:25.400+0000 7f6a5904dd80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-07-23T12:27:27.291 INFO:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2020-07-23T12:27:27.348 INFO:teuthology.orchestra.run.smithi133:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2020-07-23T12:27:27.437 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:27.428+0000 7f67d2068d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2020-07-23T12:27:27.437 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:27.428+0000 7f67d2068d80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2020-07-23T12:27:27.437 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:27.428+0000 7f67d2068d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-07-23T12:27:27.441 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:27.428+0000 7f67d2068d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-07-23T12:27:27.441 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:27.428+0000 7f67d2068d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-07-23T12:27:27.442 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:27.428+0000 7f67d2068d80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2020-07-23T12:27:27.452 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:27:27.448+0000 7f67d2068d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-07-23T12:27:29.351 INFO:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2020-07-23T12:27:29.408 INFO:tasks.ceph:Reading keys from all nodes... 2020-07-23T12:27:29.409 INFO:teuthology.orchestra.run.smithi133:> mktemp 2020-07-23T12:27:29.454 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.dDRlWfLTsE 2020-07-23T12:27:29.455 INFO:teuthology.orchestra.run.smithi133:> sudo cp /var/lib/ceph/mgr/c2-x/keyring /tmp/tmp.dDRlWfLTsE 2020-07-23T12:27:29.519 INFO:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.dDRlWfLTsE 2020-07-23T12:27:29.624 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.dDRlWfLTsE is 56B 2020-07-23T12:27:29.636 INFO:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.dDRlWfLTsE 2020-07-23T12:27:29.641 INFO:teuthology.orchestra.run.smithi133:> mktemp 2020-07-23T12:27:29.690 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.0EUemCc1Do 2020-07-23T12:27:29.691 INFO:teuthology.orchestra.run.smithi133:> sudo cp /var/lib/ceph/osd/c2-0/keyring /tmp/tmp.0EUemCc1Do 2020-07-23T12:27:29.748 INFO:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.0EUemCc1Do 2020-07-23T12:27:29.855 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.0EUemCc1Do is 56B 2020-07-23T12:27:29.907 INFO:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.0EUemCc1Do 2020-07-23T12:27:29.913 INFO:teuthology.orchestra.run.smithi133:> mktemp 2020-07-23T12:27:29.963 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.zkro72o99G 2020-07-23T12:27:29.963 INFO:teuthology.orchestra.run.smithi133:> sudo cp /var/lib/ceph/osd/c2-1/keyring /tmp/tmp.zkro72o99G 2020-07-23T12:27:30.024 INFO:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.zkro72o99G 2020-07-23T12:27:30.128 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.zkro72o99G is 56B 2020-07-23T12:27:30.140 INFO:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.zkro72o99G 2020-07-23T12:27:30.146 INFO:teuthology.orchestra.run.smithi133:> mktemp 2020-07-23T12:27:30.194 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.bRK2TtN48s 2020-07-23T12:27:30.195 INFO:teuthology.orchestra.run.smithi133:> sudo cp /var/lib/ceph/osd/c2-2/keyring /tmp/tmp.bRK2TtN48s 2020-07-23T12:27:30.251 INFO:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.bRK2TtN48s 2020-07-23T12:27:30.356 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.bRK2TtN48s is 56B 2020-07-23T12:27:30.367 INFO:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.bRK2TtN48s 2020-07-23T12:27:30.425 DEBUG:teuthology.orchestra.remote:smithi133:/etc/ceph/c2.client.0.keyring is 59B 2020-07-23T12:27:30.465 DEBUG:teuthology.orchestra.remote:smithi133:/etc/ceph/c2.client.1.keyring is 59B 2020-07-23T12:27:30.476 INFO:tasks.ceph:Adding keys to all mons... 2020-07-23T12:27:30.477 INFO:teuthology.orchestra.run.smithi133:> sudo tee -a /etc/ceph/c2.keyring 2020-07-23T12:27:30.490 INFO:teuthology.orchestra.run.smithi133.stdout:[mgr.x] 2020-07-23T12:27:30.491 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAoghlfPSRQOxAAu8PcOKBl1vyjB2EgB+KrzQ== 2020-07-23T12:27:30.491 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.0] 2020-07-23T12:27:30.491 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQArghlfJZZDFBAAZQc2BukEDTZn/ZmIg1Bqww== 2020-07-23T12:27:30.491 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.1] 2020-07-23T12:27:30.491 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAtghlfS7n9FhAAcO+lm07+Ixv7fypfUsjU0Q== 2020-07-23T12:27:30.492 INFO:teuthology.orchestra.run.smithi133.stdout:[osd.2] 2020-07-23T12:27:30.492 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQAvghlfAxKjGRAAiBz+DGb8daEyuhf588uc+Q== 2020-07-23T12:27:30.492 INFO:teuthology.orchestra.run.smithi133.stdout:[client.0] 2020-07-23T12:27:30.492 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQApghlfEvZmBBAAhQI0tU+a5ah69IngRZf0YA== 2020-07-23T12:27:30.492 INFO:teuthology.orchestra.run.smithi133.stdout:[client.1] 2020-07-23T12:27:30.493 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQApghlfoUqUCRAASRJYS6Xe9L0+Nhgu2SG64g== 2020-07-23T12:27:30.494 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-07-23T12:27:30.589 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-23T12:27:30.674 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-23T12:27:30.763 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-07-23T12:27:30.847 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-07-23T12:27:30.933 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-07-23T12:27:31.022 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-07-23T12:27:31.023 INFO:teuthology.orchestra.run.smithi133:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2020-07-23T12:27:31.080 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2020-07-23T12:27:31.180 INFO:teuthology.orchestra.run.smithi133:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2020-07-23T12:27:31.198 INFO:teuthology.orchestra.run.smithi133:> rm -- /home/ubuntu/cephtest/c2.monmap 2020-07-23T12:27:31.247 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2020-07-23T12:27:31.247 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2020-07-23T12:27:31.248 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2020-07-23T12:27:31.290 INFO:tasks.ceph.c2.mon.a:Started 2020-07-23T12:27:31.291 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2020-07-23T12:27:31.292 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2020-07-23T12:27:31.292 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2020-07-23T12:27:31.294 INFO:tasks.ceph.c2.mgr.x:Started 2020-07-23T12:27:31.296 INFO:tasks.ceph:Setting crush tunables to default 2020-07-23T12:27:31.296 INFO:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c2 osd crush tunables default 2020-07-23T12:27:31.752 INFO:teuthology.orchestra.run.smithi133.stderr:adjusted tunables profile to default 2020-07-23T12:27:31.762 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2020-07-23T12:27:31.763 INFO:teuthology.orchestra.run.smithi133:> mktemp 2020-07-23T12:27:31.770 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.8pGPjBAgq2 2020-07-23T12:27:31.770 INFO:teuthology.orchestra.run.smithi133:> sudo cp /var/lib/ceph/osd/c2-0/fsid /tmp/tmp.8pGPjBAgq2 2020-07-23T12:27:31.826 INFO:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.8pGPjBAgq2 2020-07-23T12:27:31.939 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.8pGPjBAgq2 is 37B 2020-07-23T12:27:31.996 INFO:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.8pGPjBAgq2 2020-07-23T12:27:32.004 INFO:teuthology.orchestra.run.smithi133:> mktemp 2020-07-23T12:27:32.051 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.nIdjtJvVJl 2020-07-23T12:27:32.055 INFO:teuthology.orchestra.run.smithi133:> sudo cp /var/lib/ceph/osd/c2-1/fsid /tmp/tmp.nIdjtJvVJl 2020-07-23T12:27:32.109 INFO:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.nIdjtJvVJl 2020-07-23T12:27:32.224 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.nIdjtJvVJl is 37B 2020-07-23T12:27:32.236 INFO:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.nIdjtJvVJl 2020-07-23T12:27:32.242 INFO:teuthology.orchestra.run.smithi133:> mktemp 2020-07-23T12:27:32.306 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.MP9GMhCBjz 2020-07-23T12:27:32.306 INFO:teuthology.orchestra.run.smithi133:> sudo cp /var/lib/ceph/osd/c2-2/fsid /tmp/tmp.MP9GMhCBjz 2020-07-23T12:27:32.370 INFO:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.MP9GMhCBjz 2020-07-23T12:27:32.475 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.MP9GMhCBjz is 37B 2020-07-23T12:27:32.486 INFO:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.MP9GMhCBjz 2020-07-23T12:27:32.493 INFO:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c2 osd new ae36719d-2c18-44d7-a7a2-d2db73a7b71f 0 2020-07-23T12:27:32.733 INFO:teuthology.orchestra.run.smithi133.stdout:0 2020-07-23T12:27:32.746 INFO:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c2 osd new 743af251-084c-4092-a0ce-0f8768d174a9 1 2020-07-23T12:27:32.946 INFO:teuthology.orchestra.run.smithi133.stdout:1 2020-07-23T12:27:32.958 INFO:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c2 osd new eba88b42-80a7-4bbf-94c2-3862483162aa 2 2020-07-23T12:27:33.151 INFO:teuthology.orchestra.run.smithi133.stdout:2 2020-07-23T12:27:33.162 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2020-07-23T12:27:33.163 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2020-07-23T12:27:33.167 INFO:tasks.ceph.c2.osd.0:Started 2020-07-23T12:27:33.167 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2020-07-23T12:27:33.168 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2020-07-23T12:27:33.180 INFO:tasks.ceph.c2.osd.1:Started 2020-07-23T12:27:33.187 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2020-07-23T12:27:33.188 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2020-07-23T12:27:33.190 INFO:tasks.ceph.c2.osd.2:Started 2020-07-23T12:27:33.191 INFO:tasks.ceph:Waiting for OSDs to come up 2020-07-23T12:27:33.496 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-07-23T12:27:33.666 INFO:teuthology.misc.health.smithi133.stdout: 2020-07-23T12:27:33.667 INFO:teuthology.misc.health.smithi133.stdout:{"epoch":5,"fsid":"2fb80b8c-a980-4374-ad78-15e596842194","created":"2020-07-23T12:27:31.413035+0000","modified":"2020-07-23T12:27:33.151286+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"ae36719d-2c18-44d7-a7a2-d2db73a7b71f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"743af251-084c-4092-a0ce-0f8768d174a9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eba88b42-80a7-4bbf-94c2-3862483162aa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-23T12:27:33.677 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-07-23T12:27:33.763 INFO:tasks.ceph.c2.osd.0.smithi133.stderr:2020-07-23T12:27:33.764+0000 7f7d5e214d80 -1 Falling back to public interface 2020-07-23T12:27:33.795 INFO:tasks.ceph.c2.osd.1.smithi133.stderr:2020-07-23T12:27:33.792+0000 7fd2bd524d80 -1 Falling back to public interface 2020-07-23T12:27:33.795 INFO:tasks.ceph.c2.osd.2.smithi133.stderr:2020-07-23T12:27:33.796+0000 7f411de0fd80 -1 Falling back to public interface 2020-07-23T12:27:34.178 INFO:tasks.ceph.c2.osd.1.smithi133.stderr:2020-07-23T12:27:34.180+0000 7fd2bd524d80 -1 osd.1 0 log_to_monitors {default=true} 2020-07-23T12:27:34.372 INFO:tasks.ceph.c2.osd.0.smithi133.stderr:2020-07-23T12:27:34.372+0000 7f7d5e214d80 -1 osd.0 0 log_to_monitors {default=true} 2020-07-23T12:27:34.424 INFO:tasks.ceph.c2.osd.2.smithi133.stderr:2020-07-23T12:27:34.424+0000 7f411de0fd80 -1 osd.2 0 log_to_monitors {default=true} 2020-07-23T12:27:35.452 INFO:tasks.ceph.c2.osd.2.smithi133.stderr:2020-07-23T12:27:35.448+0000 7f410785c700 -1 osd.2 0 waiting for initial osdmap 2020-07-23T12:27:35.452 INFO:tasks.ceph.c2.osd.0.smithi133.stderr:2020-07-23T12:27:35.448+0000 7f7d47c61700 -1 osd.0 0 waiting for initial osdmap 2020-07-23T12:27:35.452 INFO:tasks.ceph.c2.osd.1.smithi133.stderr:2020-07-23T12:27:35.448+0000 7fd2a6f71700 -1 osd.1 0 waiting for initial osdmap 2020-07-23T12:27:39.981 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-07-23T12:27:40.242 INFO:teuthology.misc.health.smithi133.stdout: 2020-07-23T12:27:40.243 INFO:teuthology.misc.health.smithi133.stdout:{"epoch":10,"fsid":"2fb80b8c-a980-4374-ad78-15e596842194","created":"2020-07-23T12:27:31.413035+0000","modified":"2020-07-23T12:27:38.459739+0000","last_up_change":"2020-07-23T12:27:36.441512+0000","last_in_change":"2020-07-23T12:27:36.441512+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-23T12:27:35.134927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"ae36719d-2c18-44d7-a7a2-d2db73a7b71f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":11716},{"type":"v1","addr":"172.21.15.133:6801","nonce":11716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":11716},{"type":"v1","addr":"172.21.15.133:6803","nonce":11716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":11716},{"type":"v1","addr":"172.21.15.133:6807","nonce":11716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":11716},{"type":"v1","addr":"172.21.15.133:6805","nonce":11716}]},"public_addr":"172.21.15.133:6801/11716","cluster_addr":"172.21.15.133:6803/11716","heartbeat_back_addr":"172.21.15.133:6807/11716","heartbeat_front_addr":"172.21.15.133:6805/11716","state":["exists","up"]},{"osd":1,"uuid":"743af251-084c-4092-a0ce-0f8768d174a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":11717},{"type":"v1","addr":"172.21.15.133:6809","nonce":11717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":11717},{"type":"v1","addr":"172.21.15.133:6811","nonce":11717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":11717},{"type":"v1","addr":"172.21.15.133:6815","nonce":11717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":11717},{"type":"v1","addr":"172.21.15.133:6813","nonce":11717}]},"public_addr":"172.21.15.133:6809/11717","cluster_addr":"172.21.15.133:6811/11717","heartbeat_back_addr":"172.21.15.133:6815/11717","heartbeat_front_addr":"172.21.15.133:6813/11717","state":["exists","up"]},{"osd":2,"uuid":"eba88b42-80a7-4bbf-94c2-3862483162aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":11747},{"type":"v1","addr":"172.21.15.133:6817","nonce":11747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":11747},{"type":"v1","addr":"172.21.15.133:6819","nonce":11747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":11747},{"type":"v1","addr":"172.21.15.133:6823","nonce":11747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":11747},{"type":"v1","addr":"172.21.15.133:6821","nonce":11747}]},"public_addr":"172.21.15.133:6817/11747","cluster_addr":"172.21.15.133:6819/11747","heartbeat_back_addr":"172.21.15.133:6823/11747","heartbeat_front_addr":"172.21.15.133:6821/11747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:35.408773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:35.178667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:35.434729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-23T12:27:40.255 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-07-23T12:27:40.255 INFO:tasks.ceph:Creating RBD pool 2020-07-23T12:27:40.256 INFO:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c2 osd pool create rbd 8 2020-07-23T12:27:41.490 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'rbd' created 2020-07-23T12:27:41.503 INFO:teuthology.orchestra.run.smithi133:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-07-23T12:27:42.493 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rbd' on pool 'rbd' 2020-07-23T12:27:42.505 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2020-07-23T12:27:42.506 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-07-23T12:27:42.507 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-07-23T12:27:42.759 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:42.760 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":12,"fsid":"2fb80b8c-a980-4374-ad78-15e596842194","created":"2020-07-23T12:27:31.413035+0000","modified":"2020-07-23T12:27:42.478422+0000","last_up_change":"2020-07-23T12:27:36.441512+0000","last_in_change":"2020-07-23T12:27:36.441512+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-23T12:27:35.134927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-23T12:27:40.524536+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ae36719d-2c18-44d7-a7a2-d2db73a7b71f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":11716},{"type":"v1","addr":"172.21.15.133:6801","nonce":11716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":11716},{"type":"v1","addr":"172.21.15.133:6803","nonce":11716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":11716},{"type":"v1","addr":"172.21.15.133:6807","nonce":11716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":11716},{"type":"v1","addr":"172.21.15.133:6805","nonce":11716}]},"public_addr":"172.21.15.133:6801/11716","cluster_addr":"172.21.15.133:6803/11716","heartbeat_back_addr":"172.21.15.133:6807/11716","heartbeat_front_addr":"172.21.15.133:6805/11716","state":["exists","up"]},{"osd":1,"uuid":"743af251-084c-4092-a0ce-0f8768d174a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":11717},{"type":"v1","addr":"172.21.15.133:6809","nonce":11717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":11717},{"type":"v1","addr":"172.21.15.133:6811","nonce":11717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":11717},{"type":"v1","addr":"172.21.15.133:6815","nonce":11717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":11717},{"type":"v1","addr":"172.21.15.133:6813","nonce":11717}]},"public_addr":"172.21.15.133:6809/11717","cluster_addr":"172.21.15.133:6811/11717","heartbeat_back_addr":"172.21.15.133:6815/11717","heartbeat_front_addr":"172.21.15.133:6813/11717","state":["exists","up"]},{"osd":2,"uuid":"eba88b42-80a7-4bbf-94c2-3862483162aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":11747},{"type":"v1","addr":"172.21.15.133:6817","nonce":11747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":11747},{"type":"v1","addr":"172.21.15.133:6819","nonce":11747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":11747},{"type":"v1","addr":"172.21.15.133:6823","nonce":11747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":11747},{"type":"v1","addr":"172.21.15.133:6821","nonce":11747}]},"public_addr":"172.21.15.133:6817/11747","cluster_addr":"172.21.15.133:6819/11747","heartbeat_back_addr":"172.21.15.133:6823/11747","heartbeat_front_addr":"172.21.15.133:6821/11747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:35.408773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:35.178667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:35.434729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-23T12:27:42.773 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-07-23T12:27:35.134927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-07-23T12:27:40.524536+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-07-23T12:27:42.774 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get device_health_metrics pg_num 2020-07-23T12:27:43.043 INFO:teuthology.orchestra.run.smithi133.stdout:pg_num: 1 2020-07-23T12:27:43.063 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get rbd pg_num 2020-07-23T12:27:43.347 INFO:teuthology.orchestra.run.smithi133.stdout:pg_num: 8 2020-07-23T12:27:43.357 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2020-07-23T12:27:43.357 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2020-07-23T12:27:43.357 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early mgr dump --format=json 2020-07-23T12:27:43.655 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:43.667 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":11508},{"type":"v1","addr":"172.21.15.133:6825","nonce":11508}]},"active_addr":"172.21.15.133:6825/11508","active_change":"2020-07-23T12:27:35.106360+0000","active_mgr_features":4540138292836696063,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":279623724}]},{"addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":2793610639}]},{"addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":1196362718}]}]}} 2020-07-23T12:27:43.670 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2020-07-23T12:27:43.670 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2020-07-23T12:27:43.671 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-07-23T12:27:43.924 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:43.925 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":13,"fsid":"2fb80b8c-a980-4374-ad78-15e596842194","created":"2020-07-23T12:27:31.413035+0000","modified":"2020-07-23T12:27:43.486310+0000","last_up_change":"2020-07-23T12:27:36.441512+0000","last_in_change":"2020-07-23T12:27:36.441512+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-23T12:27:35.134927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-23T12:27:40.524536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ae36719d-2c18-44d7-a7a2-d2db73a7b71f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":11716},{"type":"v1","addr":"172.21.15.133:6801","nonce":11716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":11716},{"type":"v1","addr":"172.21.15.133:6803","nonce":11716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":11716},{"type":"v1","addr":"172.21.15.133:6807","nonce":11716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":11716},{"type":"v1","addr":"172.21.15.133:6805","nonce":11716}]},"public_addr":"172.21.15.133:6801/11716","cluster_addr":"172.21.15.133:6803/11716","heartbeat_back_addr":"172.21.15.133:6807/11716","heartbeat_front_addr":"172.21.15.133:6805/11716","state":["exists","up"]},{"osd":1,"uuid":"743af251-084c-4092-a0ce-0f8768d174a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":11717},{"type":"v1","addr":"172.21.15.133:6809","nonce":11717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":11717},{"type":"v1","addr":"172.21.15.133:6811","nonce":11717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":11717},{"type":"v1","addr":"172.21.15.133:6815","nonce":11717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":11717},{"type":"v1","addr":"172.21.15.133:6813","nonce":11717}]},"public_addr":"172.21.15.133:6809/11717","cluster_addr":"172.21.15.133:6811/11717","heartbeat_back_addr":"172.21.15.133:6815/11717","heartbeat_front_addr":"172.21.15.133:6813/11717","state":["exists","up"]},{"osd":2,"uuid":"eba88b42-80a7-4bbf-94c2-3862483162aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":11747},{"type":"v1","addr":"172.21.15.133:6817","nonce":11747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":11747},{"type":"v1","addr":"172.21.15.133:6819","nonce":11747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":11747},{"type":"v1","addr":"172.21.15.133:6823","nonce":11747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":11747},{"type":"v1","addr":"172.21.15.133:6821","nonce":11747}]},"public_addr":"172.21.15.133:6817/11747","cluster_addr":"172.21.15.133:6819/11747","heartbeat_back_addr":"172.21.15.133:6823/11747","heartbeat_front_addr":"172.21.15.133:6821/11747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:35.408773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:35.178667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:35.434729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-23T12:27:43.944 INFO:tasks.ceph.ceph_manager.c2:all up! 2020-07-23T12:27:43.945 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-07-23T12:27:44.210 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:44.210 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":13,"fsid":"2fb80b8c-a980-4374-ad78-15e596842194","created":"2020-07-23T12:27:31.413035+0000","modified":"2020-07-23T12:27:43.486310+0000","last_up_change":"2020-07-23T12:27:36.441512+0000","last_in_change":"2020-07-23T12:27:36.441512+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-07-23T12:27:35.134927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-07-23T12:27:40.524536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ae36719d-2c18-44d7-a7a2-d2db73a7b71f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":11716},{"type":"v1","addr":"172.21.15.133:6801","nonce":11716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":11716},{"type":"v1","addr":"172.21.15.133:6803","nonce":11716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":11716},{"type":"v1","addr":"172.21.15.133:6807","nonce":11716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":11716},{"type":"v1","addr":"172.21.15.133:6805","nonce":11716}]},"public_addr":"172.21.15.133:6801/11716","cluster_addr":"172.21.15.133:6803/11716","heartbeat_back_addr":"172.21.15.133:6807/11716","heartbeat_front_addr":"172.21.15.133:6805/11716","state":["exists","up"]},{"osd":1,"uuid":"743af251-084c-4092-a0ce-0f8768d174a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":11717},{"type":"v1","addr":"172.21.15.133:6809","nonce":11717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":11717},{"type":"v1","addr":"172.21.15.133:6811","nonce":11717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":11717},{"type":"v1","addr":"172.21.15.133:6815","nonce":11717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":11717},{"type":"v1","addr":"172.21.15.133:6813","nonce":11717}]},"public_addr":"172.21.15.133:6809/11717","cluster_addr":"172.21.15.133:6811/11717","heartbeat_back_addr":"172.21.15.133:6815/11717","heartbeat_front_addr":"172.21.15.133:6813/11717","state":["exists","up"]},{"osd":2,"uuid":"eba88b42-80a7-4bbf-94c2-3862483162aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":11747},{"type":"v1","addr":"172.21.15.133:6817","nonce":11747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":11747},{"type":"v1","addr":"172.21.15.133:6819","nonce":11747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":11747},{"type":"v1","addr":"172.21.15.133:6823","nonce":11747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":11747},{"type":"v1","addr":"172.21.15.133:6821","nonce":11747}]},"public_addr":"172.21.15.133:6817/11747","cluster_addr":"172.21.15.133:6819/11747","heartbeat_back_addr":"172.21.15.133:6823/11747","heartbeat_front_addr":"172.21.15.133:6821/11747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:35.408773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:35.178667+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-07-23T12:27:35.434729+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-07-23T12:27:44.246 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.0 flush_pg_stats 2020-07-23T12:27:44.400 INFO:teuthology.orchestra.run.smithi133.stdout:34359738371 2020-07-23T12:27:44.401 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.1 flush_pg_stats 2020-07-23T12:27:44.563 INFO:teuthology.orchestra.run.smithi133.stdout:34359738372 2020-07-23T12:27:44.564 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.2 flush_pg_stats 2020-07-23T12:27:44.728 INFO:teuthology.orchestra.run.smithi133.stdout:34359738371 2020-07-23T12:27:44.729 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-07-23T12:27:44.995 INFO:teuthology.orchestra.run.smithi133.stdout:34359738370 2020-07-23T12:27:45.008 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2020-07-23T12:27:46.009 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-07-23T12:27:46.276 INFO:teuthology.orchestra.run.smithi133.stdout:34359738371 2020-07-23T12:27:46.289 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.0 2020-07-23T12:27:46.290 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.1 2020-07-23T12:27:46.548 INFO:teuthology.orchestra.run.smithi133.stdout:34359738372 2020-07-23T12:27:46.562 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.1 2020-07-23T12:27:46.563 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.2 2020-07-23T12:27:46.827 INFO:teuthology.orchestra.run.smithi133.stdout:34359738371 2020-07-23T12:27:46.840 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.2 2020-07-23T12:27:46.840 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2020-07-23T12:27:46.841 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-07-23T12:27:47.103 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:47.103 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2020-07-23T12:27:47.111 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-07-23T12:27:45.115686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.634479"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.504511+0000","last_change":"2020-07-23T12:27:42.497892+0000","last_active":"2020-07-23T12:27:42.504511+0000","last_peered":"2020-07-23T12:27:42.504511+0000","last_clean":"2020-07-23T12:27:42.504511+0000","last_became_active":"2020-07-23T12:27:42.497433+0000","last_became_peered":"2020-07-23T12:27:42.497433+0000","last_unstale":"2020-07-23T12:27:42.504511+0000","last_undegraded":"2020-07-23T12:27:42.504511+0000","last_fullsized":"2020-07-23T12:27:42.504511+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_clean_scrub_stamp":"2020-07-23T12:27:41.473718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.504506+0000","last_change":"2020-07-23T12:27:42.498283+0000","last_active":"2020-07-23T12:27:42.504506+0000","last_peered":"2020-07-23T12:27:42.504506+0000","last_clean":"2020-07-23T12:27:42.504506+0000","last_became_active":"2020-07-23T12:27:42.497756+0000","last_became_peered":"2020-07-23T12:27:42.497756+0000","last_unstale":"2020-07-23T12:27:42.504506+0000","last_undegraded":"2020-07-23T12:27:42.504506+0000","last_fullsized":"2020-07-23T12:27:42.504506+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_clean_scrub_stamp":"2020-07-23T12:27:41.473718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.504921+0000","last_change":"2020-07-23T12:27:42.498796+0000","last_active":"2020-07-23T12:27:42.504921+0000","last_peered":"2020-07-23T12:27:42.504921+0000","last_clean":"2020-07-23T12:27:42.504921+0000","last_became_active":"2020-07-23T12:27:42.498005+0000","last_became_peered":"2020-07-23T12:27:42.498005+0000","last_unstale":"2020-07-23T12:27:42.504921+0000","last_undegraded":"2020-07-23T12:27:42.504921+0000","last_fullsized":"2020-07-23T12:27:42.504921+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_clean_scrub_stamp":"2020-07-23T12:27:41.473718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.494799+0000","last_change":"2020-07-23T12:27:42.494799+0000","last_active":"2020-07-23T12:27:42.494799+0000","last_peered":"2020-07-23T12:27:42.494799+0000","last_clean":"2020-07-23T12:27:42.494799+0000","last_became_active":"2020-07-23T12:27:42.494444+0000","last_became_peered":"2020-07-23T12:27:42.494444+0000","last_unstale":"2020-07-23T12:27:42.494799+0000","last_undegraded":"2020-07-23T12:27:42.494799+0000","last_fullsized":"2020-07-23T12:27:42.494799+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_clean_scrub_stamp":"2020-07-23T12:27:41.473718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.490863+0000","last_change":"2020-07-23T12:27:37.686977+0000","last_active":"2020-07-23T12:27:42.490863+0000","last_peered":"2020-07-23T12:27:42.490863+0000","last_clean":"2020-07-23T12:27:42.490863+0000","last_became_active":"2020-07-23T12:27:37.685507+0000","last_became_peered":"2020-07-23T12:27:37.685507+0000","last_unstale":"2020-07-23T12:27:42.490863+0000","last_undegraded":"2020-07-23T12:27:42.490863+0000","last_fullsized":"2020-07-23T12:27:42.490863+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:35.437853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:35.437853+0000","last_clean_scrub_stamp":"2020-07-23T12:27:35.437853+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.504962+0000","last_change":"2020-07-23T12:27:42.499800+0000","last_active":"2020-07-23T12:27:42.504962+0000","last_peered":"2020-07-23T12:27:42.504962+0000","last_clean":"2020-07-23T12:27:42.504962+0000","last_became_active":"2020-07-23T12:27:42.499402+0000","last_became_peered":"2020-07-23T12:27:42.499402+0000","last_unstale":"2020-07-23T12:27:42.504962+0000","last_undegraded":"2020-07-23T12:27:42.504962+0000","last_fullsized":"2020-07-23T12:27:42.504962+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_clean_scrub_stamp":"2020-07-23T12:27:41.473718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.500352+0000","last_change":"2020-07-23T12:27:42.500352+0000","last_active":"2020-07-23T12:27:42.500352+0000","last_peered":"2020-07-23T12:27:42.500352+0000","last_clean":"2020-07-23T12:27:42.500352+0000","last_became_active":"2020-07-23T12:27:42.499962+0000","last_became_peered":"2020-07-23T12:27:42.499962+0000","last_unstale":"2020-07-23T12:27:42.500352+0000","last_undegraded":"2020-07-23T12:27:42.500352+0000","last_fullsized":"2020-07-23T12:27:42.500352+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_clean_scrub_stamp":"2020-07-23T12:27:41.473718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.501303+0000","last_change":"2020-07-23T12:27:42.501303+0000","last_active":"2020-07-23T12:27:42.501303+0000","last_peered":"2020-07-23T12:27:42.501303+0000","last_clean":"2020-07-23T12:27:42.501303+0000","last_became_active":"2020-07-23T12:27:42.500682+0000","last_became_peered":"2020-07-23T12:27:42.500682+0000","last_unstale":"2020-07-23T12:27:42.501303+0000","last_undegraded":"2020-07-23T12:27:42.501303+0000","last_fullsized":"2020-07-23T12:27:42.501303+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_clean_scrub_stamp":"2020-07-23T12:27:41.473718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.504527+0000","last_change":"2020-07-23T12:27:42.497673+0000","last_active":"2020-07-23T12:27:42.504527+0000","last_peered":"2020-07-23T12:27:42.504527+0000","last_clean":"2020-07-23T12:27:42.504527+0000","last_became_active":"2020-07-23T12:27:42.497201+0000","last_became_peered":"2020-07-23T12:27:42.497201+0000","last_unstale":"2020-07-23T12:27:42.504527+0000","last_undegraded":"2020-07-23T12:27:42.504527+0000","last_fullsized":"2020-07-23T12:27:42.504527+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_clean_scrub_stamp":"2020-07-23T12:27:41.473718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.321}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4019999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.482}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7729999999999999}]}]}]}} 2020-07-23T12:27:47.113 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-07-23T12:27:47.370 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:47.372 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2020-07-23T12:27:47.382 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-07-23T12:27:47.116245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.635038"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.504511+0000","last_change":"2020-07-23T12:27:42.497892+0000","last_active":"2020-07-23T12:27:42.504511+0000","last_peered":"2020-07-23T12:27:42.504511+0000","last_clean":"2020-07-23T12:27:42.504511+0000","last_became_active":"2020-07-23T12:27:42.497433+0000","last_became_peered":"2020-07-23T12:27:42.497433+0000","last_unstale":"2020-07-23T12:27:42.504511+0000","last_undegraded":"2020-07-23T12:27:42.504511+0000","last_fullsized":"2020-07-23T12:27:42.504511+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_clean_scrub_stamp":"2020-07-23T12:27:41.473718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.504506+0000","last_change":"2020-07-23T12:27:42.498283+0000","last_active":"2020-07-23T12:27:42.504506+0000","last_peered":"2020-07-23T12:27:42.504506+0000","last_clean":"2020-07-23T12:27:42.504506+0000","last_became_active":"2020-07-23T12:27:42.497756+0000","last_became_peered":"2020-07-23T12:27:42.497756+0000","last_unstale":"2020-07-23T12:27:42.504506+0000","last_undegraded":"2020-07-23T12:27:42.504506+0000","last_fullsized":"2020-07-23T12:27:42.504506+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_clean_scrub_stamp":"2020-07-23T12:27:41.473718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.504921+0000","last_change":"2020-07-23T12:27:42.498796+0000","last_active":"2020-07-23T12:27:42.504921+0000","last_peered":"2020-07-23T12:27:42.504921+0000","last_clean":"2020-07-23T12:27:42.504921+0000","last_became_active":"2020-07-23T12:27:42.498005+0000","last_became_peered":"2020-07-23T12:27:42.498005+0000","last_unstale":"2020-07-23T12:27:42.504921+0000","last_undegraded":"2020-07-23T12:27:42.504921+0000","last_fullsized":"2020-07-23T12:27:42.504921+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_clean_scrub_stamp":"2020-07-23T12:27:41.473718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.494799+0000","last_change":"2020-07-23T12:27:42.494799+0000","last_active":"2020-07-23T12:27:42.494799+0000","last_peered":"2020-07-23T12:27:42.494799+0000","last_clean":"2020-07-23T12:27:42.494799+0000","last_became_active":"2020-07-23T12:27:42.494444+0000","last_became_peered":"2020-07-23T12:27:42.494444+0000","last_unstale":"2020-07-23T12:27:42.494799+0000","last_undegraded":"2020-07-23T12:27:42.494799+0000","last_fullsized":"2020-07-23T12:27:42.494799+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_clean_scrub_stamp":"2020-07-23T12:27:41.473718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.490863+0000","last_change":"2020-07-23T12:27:37.686977+0000","last_active":"2020-07-23T12:27:42.490863+0000","last_peered":"2020-07-23T12:27:42.490863+0000","last_clean":"2020-07-23T12:27:42.490863+0000","last_became_active":"2020-07-23T12:27:37.685507+0000","last_became_peered":"2020-07-23T12:27:37.685507+0000","last_unstale":"2020-07-23T12:27:42.490863+0000","last_undegraded":"2020-07-23T12:27:42.490863+0000","last_fullsized":"2020-07-23T12:27:42.490863+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:35.437853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:35.437853+0000","last_clean_scrub_stamp":"2020-07-23T12:27:35.437853+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.504962+0000","last_change":"2020-07-23T12:27:42.499800+0000","last_active":"2020-07-23T12:27:42.504962+0000","last_peered":"2020-07-23T12:27:42.504962+0000","last_clean":"2020-07-23T12:27:42.504962+0000","last_became_active":"2020-07-23T12:27:42.499402+0000","last_became_peered":"2020-07-23T12:27:42.499402+0000","last_unstale":"2020-07-23T12:27:42.504962+0000","last_undegraded":"2020-07-23T12:27:42.504962+0000","last_fullsized":"2020-07-23T12:27:42.504962+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_clean_scrub_stamp":"2020-07-23T12:27:41.473718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.500352+0000","last_change":"2020-07-23T12:27:42.500352+0000","last_active":"2020-07-23T12:27:42.500352+0000","last_peered":"2020-07-23T12:27:42.500352+0000","last_clean":"2020-07-23T12:27:42.500352+0000","last_became_active":"2020-07-23T12:27:42.499962+0000","last_became_peered":"2020-07-23T12:27:42.499962+0000","last_unstale":"2020-07-23T12:27:42.500352+0000","last_undegraded":"2020-07-23T12:27:42.500352+0000","last_fullsized":"2020-07-23T12:27:42.500352+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_clean_scrub_stamp":"2020-07-23T12:27:41.473718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.501303+0000","last_change":"2020-07-23T12:27:42.501303+0000","last_active":"2020-07-23T12:27:42.501303+0000","last_peered":"2020-07-23T12:27:42.501303+0000","last_clean":"2020-07-23T12:27:42.501303+0000","last_became_active":"2020-07-23T12:27:42.500682+0000","last_became_peered":"2020-07-23T12:27:42.500682+0000","last_unstale":"2020-07-23T12:27:42.501303+0000","last_undegraded":"2020-07-23T12:27:42.501303+0000","last_fullsized":"2020-07-23T12:27:42.501303+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_clean_scrub_stamp":"2020-07-23T12:27:41.473718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-07-23T12:27:42.504527+0000","last_change":"2020-07-23T12:27:42.497673+0000","last_active":"2020-07-23T12:27:42.504527+0000","last_peered":"2020-07-23T12:27:42.504527+0000","last_clean":"2020-07-23T12:27:42.504527+0000","last_became_active":"2020-07-23T12:27:42.497201+0000","last_became_peered":"2020-07-23T12:27:42.497201+0000","last_unstale":"2020-07-23T12:27:42.504527+0000","last_undegraded":"2020-07-23T12:27:42.504527+0000","last_fullsized":"2020-07-23T12:27:42.504527+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-07-23T12:27:41.473718+0000","last_clean_scrub_stamp":"2020-07-23T12:27:41.473718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.321}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4019999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.482}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7729999999999999}]}]}]}} 2020-07-23T12:27:47.384 INFO:tasks.ceph.ceph_manager.c2:clean! 2020-07-23T12:27:47.384 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2020-07-23T12:27:47.384 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2020-07-23T12:27:47.385 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early health --format=json 2020-07-23T12:27:47.688 INFO:teuthology.orchestra.run.smithi133.stdout: 2020-07-23T12:27:47.700 INFO:teuthology.orchestra.run.smithi133.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-07-23T12:27:47.701 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2020-07-23T12:27:47.701 INFO:teuthology.run_tasks:Running task rgw... 2020-07-23T12:27:47.732 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2020-07-23T12:27:47.733 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2020-07-23T12:27:47.733 INFO:tasks.rgw:Creating data pools 2020-07-23T12:27:47.733 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2020-07-23T12:27:47.734 INFO:teuthology.orchestra.run.smithi137:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-07-23T12:27:48.962 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'default.rgw.buckets.data' created 2020-07-23T12:27:48.977 INFO:teuthology.orchestra.run.smithi137:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-07-23T12:27:49.975 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-07-23T12:27:49.997 INFO:teuthology.orchestra.run.smithi137:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-07-23T12:27:50.969 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'default.rgw.buckets.index' created 2020-07-23T12:27:50.997 INFO:teuthology.orchestra.run.smithi137:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-07-23T12:27:51.983 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-07-23T12:27:52.008 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2020-07-23T12:27:52.009 INFO:teuthology.orchestra.run.smithi137:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-07-23T12:27:52.262 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'default.rgw.buckets.data' already exists 2020-07-23T12:27:52.272 INFO:teuthology.orchestra.run.smithi137:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-07-23T12:27:52.989 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-07-23T12:27:53.003 INFO:teuthology.orchestra.run.smithi137:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-07-23T12:27:53.280 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'default.rgw.buckets.index' already exists 2020-07-23T12:27:53.289 INFO:teuthology.orchestra.run.smithi137:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-07-23T12:27:53.997 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-07-23T12:27:54.016 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2020-07-23T12:27:54.017 INFO:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-07-23T12:27:54.408 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'default.rgw.buckets.data' created 2020-07-23T12:27:54.423 INFO:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-07-23T12:27:55.443 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-07-23T12:27:55.458 INFO:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-07-23T12:27:56.433 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'default.rgw.buckets.index' created 2020-07-23T12:27:56.448 INFO:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-07-23T12:27:57.453 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-07-23T12:27:57.472 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2020-07-23T12:27:57.473 INFO:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-07-23T12:27:57.769 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'default.rgw.buckets.data' already exists 2020-07-23T12:27:57.779 INFO:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-07-23T12:27:58.429 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-07-23T12:27:58.443 INFO:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-07-23T12:27:58.714 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'default.rgw.buckets.index' already exists 2020-07-23T12:27:58.725 INFO:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-07-23T12:27:59.441 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-07-23T12:27:59.452 DEBUG:tasks.rgw:Pools created 2020-07-23T12:27:59.453 INFO:tasks.rgw:Configuring compression type = random 2020-07-23T12:27:59.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2020-07-23T12:27:59.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2020-07-23T12:27:59.456 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2020-07-23T12:27:59.493 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:27:59.494 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:27:59.505 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:27:59.502+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:00.936 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.930+0000 7f83cdc7c080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:00.937 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.930+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:00.937 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.934+0000 7f83cdc7c080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:00.937 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.934+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:00.938 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.934+0000 7f83cdc7c080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:00.938 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.934+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:00.938 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.934+0000 7f83cdc7c080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:00.957 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.954+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:00.958 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.954+0000 7f83cdc7c080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:00.958 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.954+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:00.960 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.954+0000 7f83cdc7c080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:00.960 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.954+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:00.961 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.954+0000 7f83cdc7c080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:00.961 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.954+0000 7f83cdc7c080 10 cannot find current period zonegroup using local zonegroup 2020-07-23T12:28:00.961 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.954+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:00.961 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.954+0000 7f83cdc7c080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:00.962 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.954+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:00.962 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.954+0000 7f83cdc7c080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:00.962 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.954+0000 7f83cdc7c080 10 Creating default zonegroup 2020-07-23T12:28:00.962 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.954+0000 7f83cdc7c080 20 get_system_obj_state: rctx=0x7ffde4fa5e30 obj=:.pools.avail state=0x55d3f808be40 s->prefetch_data=0 2020-07-23T12:28:00.962 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.958+0000 7f83cdc7c080 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-07-23T12:28:00.974 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.970+0000 7f83cdc7c080 10 WARNING: store->list_zones() returned r=-2 2020-07-23T12:28:00.975 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.970+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:00.975 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.970+0000 7f83cdc7c080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:00.980 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.978+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:00.981 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.978+0000 7f83cdc7c080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:00.981 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.978+0000 7f83cdc7c080 10 could not read realm id: (2) No such file or directory 2020-07-23T12:28:00.981 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.978+0000 7f83cdc7c080 10 WARNING: failed to set zone as default, r=-22 2020-07-23T12:28:00.982 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.978+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:00.982 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.978+0000 7f83cdc7c080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:00.987 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.986+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:00.989 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.986+0000 7f83cdc7c080 20 rados_obj.operate() r=0 bl.length=805 2020-07-23T12:28:00.989 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.986+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:00.990 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.986+0000 7f83cdc7c080 20 rados_obj.operate() r=0 bl.length=358 2020-07-23T12:28:00.990 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.986+0000 7f83cdc7c080 20 zonegroup default 2020-07-23T12:28:00.991 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.986+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:00.992 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.986+0000 7f83cdc7c080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:00.992 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.986+0000 7f83cdc7c080 10 Cannot find current period zone using local zone 2020-07-23T12:28:00.993 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.986+0000 7f83cdc7c080 10 Using default name default 2020-07-23T12:28:00.993 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.986+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:00.993 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.986+0000 7f83cdc7c080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:00.993 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.986+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:00.993 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.990+0000 7f83cdc7c080 20 rados_obj.operate() r=0 bl.length=805 2020-07-23T12:28:00.994 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.990+0000 7f83cdc7c080 20 zone default found 2020-07-23T12:28:00.994 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:00.990+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:02.964 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:02.961+0000 7f83cdc7c080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:02.965 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:02.961+0000 7f83cdc7c080 20 rados->read ofs=0 len=0 2020-07-23T12:28:02.965 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:02.965+0000 7f83cdc7c080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:02.965 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:02.965+0000 7f83cdc7c080 20 started sync module instance, tier type = 2020-07-23T12:28:02.965 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:02.965+0000 7f83cdc7c080 20 started zone id=7b7a41ba-6236-454d-8508-c57d66101f1b (name=default) with tier type = 2020-07-23T12:28:05.017 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:05.016+0000 7f83cdc7c080 20 add_watcher() i=0 2020-07-23T12:28:05.018 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:05.016+0000 7f83cdc7c080 20 add_watcher() i=1 2020-07-23T12:28:05.018 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:05.016+0000 7f83cdc7c080 20 add_watcher() i=2 2020-07-23T12:28:05.018 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:05.016+0000 7f83cdc7c080 20 add_watcher() i=3 2020-07-23T12:28:05.018 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:05.016+0000 7f83cdc7c080 20 add_watcher() i=4 2020-07-23T12:28:05.019 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:05.016+0000 7f83cdc7c080 20 add_watcher() i=5 2020-07-23T12:28:05.019 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:05.016+0000 7f83cdc7c080 20 add_watcher() i=6 2020-07-23T12:28:05.020 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:05.016+0000 7f83cdc7c080 20 add_watcher() i=7 2020-07-23T12:28:05.020 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:05.016+0000 7f83cdc7c080 2 all 8 watchers are set, enabling cache 2020-07-23T12:28:05.020 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:05.016+0000 7f83cdc7c080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-23T12:28:05.020 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:05.016+0000 7f83cdc7c080 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-23T12:28:05.020 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:05.016+0000 7f8389feb700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-23T12:28:08.966 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:08.959+0000 7f83cdc7c080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-07-23T12:28:09.137 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.131+0000 7f82effff700 20 reqs_thread_entry: start 2020-07-23T12:28:09.138 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.131+0000 7f83cdc7c080 20 init_complete bucket index max shards: 11 2020-07-23T12:28:09.138 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.131+0000 7f82ee7fc700 20 reqs_thread_entry: start 2020-07-23T12:28:09.139 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.131+0000 7f82ecff9700 20 reqs_thread_entry: start 2020-07-23T12:28:09.166 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.159+0000 7f8389feb700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-23T12:28:09.168 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.163+0000 7f83cdc7c080 20 remove_watcher() i=0 2020-07-23T12:28:09.170 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.163+0000 7f83cdc7c080 2 removed watcher, disabling cache 2020-07-23T12:28:09.171 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.167+0000 7f83cdc7c080 20 remove_watcher() i=1 2020-07-23T12:28:09.177 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.171+0000 7f83cdc7c080 20 remove_watcher() i=2 2020-07-23T12:28:09.180 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.175+0000 7f83cdc7c080 20 remove_watcher() i=3 2020-07-23T12:28:09.182 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.179+0000 7f83cdc7c080 20 remove_watcher() i=4 2020-07-23T12:28:09.185 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.179+0000 7f83cdc7c080 20 remove_watcher() i=5 2020-07-23T12:28:09.188 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.183+0000 7f83cdc7c080 20 remove_watcher() i=6 2020-07-23T12:28:09.191 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.187+0000 7f83cdc7c080 20 remove_watcher() i=7 2020-07-23T12:28:09.219 INFO:teuthology.orchestra.run.smithi137.stdout:[] 2020-07-23T12:28:09.219 DEBUG:tasks.util.rgw: json result: [] 2020-07-23T12:28:09.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-23T12:28:09.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-23T12:28:09.221 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-07-23T12:28:09.301 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:09.301 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:09.312 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.307+0000 7ff63e316080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-23T12:28:09.313 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.307+0000 7ff63e316080 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-23T12:28:09.319 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.315+0000 7ff63e316080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.319 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.315+0000 7ff571ffb700 20 reqs_thread_entry: start 2020-07-23T12:28:09.321 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.315+0000 7ff63e316080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:09.321 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.315+0000 7ff63e316080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.321 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.315+0000 7ff63e316080 20 rados_obj.operate() r=0 bl.length=805 2020-07-23T12:28:09.321 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.315+0000 7ff63e316080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.322 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.315+0000 7ff63e316080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:09.322 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.315+0000 7ff63e316080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.323 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.315+0000 7ff63e316080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:09.323 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.315+0000 7ff63e316080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.323 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.319+0000 7ff63e316080 20 rados_obj.operate() r=0 bl.length=358 2020-07-23T12:28:09.333 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"7b7a41ba-6236-454d-8508-c57d66101f1b","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-07-23T12:28:09.333 DEBUG:tasks.util.rgw: json result: {'id': '7b7a41ba-6236-454d-8508-c57d66101f1b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-07-23T12:28:09.334 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2020-07-23T12:28:09.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2020-07-23T12:28:09.335 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2020-07-23T12:28:09.409 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:09.409 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:09.421 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.415+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.424 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.419+0000 7fb93d0bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:09.424 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.419+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.424 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.419+0000 7fb93d0bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:09.425 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.419+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.425 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.419+0000 7fb93d0bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:09.425 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.419+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.426 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.419+0000 7fb93d0bb080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:09.451 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.447+0000 7fb93d0bb080 20 RGWRados::pool_iterate: got zone_info.7b7a41ba-6236-454d-8508-c57d66101f1b 2020-07-23T12:28:09.452 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.447+0000 7fb93d0bb080 20 RGWRados::pool_iterate: got zone_names.default 2020-07-23T12:28:09.452 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.447+0000 7fb93d0bb080 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-07-23T12:28:09.453 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.447+0000 7fb93d0bb080 20 RGWRados::pool_iterate: got zonegroup_info.b46a0e1a-8314-4535-815e-cac0d7837296 2020-07-23T12:28:09.453 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.447+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.455 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.447+0000 7fb93d0bb080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:09.455 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.447+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.455 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.447+0000 7fb93d0bb080 20 rados_obj.operate() r=0 bl.length=821 2020-07-23T12:28:09.456 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.447+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.456 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.447+0000 7fb93d0bb080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:09.456 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.447+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.457 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.451+0000 7fb93d0bb080 20 rados_obj.operate() r=0 bl.length=358 2020-07-23T12:28:09.457 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.451+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.459 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.451+0000 7fb93d0bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:09.459 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.451+0000 7fb93d0bb080 10 cannot find current period zonegroup using local zonegroup 2020-07-23T12:28:09.460 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.451+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.460 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.451+0000 7fb93d0bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:09.460 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.451+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.461 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.451+0000 7fb93d0bb080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:09.461 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.451+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.461 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.451+0000 7fb93d0bb080 20 rados_obj.operate() r=0 bl.length=358 2020-07-23T12:28:09.461 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.451+0000 7fb93d0bb080 20 zonegroup default 2020-07-23T12:28:09.461 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.451+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.462 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.451+0000 7fb93d0bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:09.462 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.451+0000 7fb93d0bb080 10 Cannot find current period zone using local zone 2020-07-23T12:28:09.462 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.451+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.462 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.455+0000 7fb93d0bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:09.462 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.455+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.464 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.455+0000 7fb93d0bb080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:09.464 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.455+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.464 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.455+0000 7fb93d0bb080 20 rados_obj.operate() r=0 bl.length=821 2020-07-23T12:28:09.464 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.455+0000 7fb93d0bb080 20 zone default found 2020-07-23T12:28:09.464 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.455+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.465 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.455+0000 7fb93d0bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:09.465 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.455+0000 7fb93d0bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.465 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.455+0000 7fb93d0bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:09.465 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.455+0000 7fb93d0bb080 20 started sync module instance, tier type = 2020-07-23T12:28:09.465 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.455+0000 7fb93d0bb080 20 started zone id=7b7a41ba-6236-454d-8508-c57d66101f1b (name=default) with tier type = 2020-07-23T12:28:09.492 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.487+0000 7fb93d0bb080 20 add_watcher() i=0 2020-07-23T12:28:09.493 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.487+0000 7fb93d0bb080 20 add_watcher() i=1 2020-07-23T12:28:09.493 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.487+0000 7fb93d0bb080 20 add_watcher() i=2 2020-07-23T12:28:09.494 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.487+0000 7fb93d0bb080 20 add_watcher() i=3 2020-07-23T12:28:09.494 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.487+0000 7fb93d0bb080 20 add_watcher() i=4 2020-07-23T12:28:09.494 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.487+0000 7fb93d0bb080 20 add_watcher() i=5 2020-07-23T12:28:09.494 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.487+0000 7fb93d0bb080 20 add_watcher() i=6 2020-07-23T12:28:09.495 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.487+0000 7fb93d0bb080 20 add_watcher() i=7 2020-07-23T12:28:09.495 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.487+0000 7fb93d0bb080 2 all 8 watchers are set, enabling cache 2020-07-23T12:28:09.496 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.487+0000 7fb93d0bb080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-23T12:28:09.496 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.487+0000 7fb93d0bb080 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-23T12:28:09.496 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.487+0000 7fb8f9fdb700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-23T12:28:09.591 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.587+0000 7fb86bfff700 20 reqs_thread_entry: start 2020-07-23T12:28:09.592 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.587+0000 7fb869ffb700 20 reqs_thread_entry: start 2020-07-23T12:28:09.592 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.587+0000 7fb93d0bb080 20 init_complete bucket index max shards: 11 2020-07-23T12:28:09.593 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.587+0000 7fb857fff700 20 reqs_thread_entry: start 2020-07-23T12:28:09.617 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.611+0000 7fb8f9fdb700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-23T12:28:09.620 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.615+0000 7fb93d0bb080 20 remove_watcher() i=0 2020-07-23T12:28:09.620 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.615+0000 7fb93d0bb080 2 removed watcher, disabling cache 2020-07-23T12:28:09.623 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.619+0000 7fb93d0bb080 20 remove_watcher() i=1 2020-07-23T12:28:09.626 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.619+0000 7fb93d0bb080 20 remove_watcher() i=2 2020-07-23T12:28:09.632 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.623+0000 7fb93d0bb080 20 remove_watcher() i=3 2020-07-23T12:28:09.634 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.627+0000 7fb93d0bb080 20 remove_watcher() i=4 2020-07-23T12:28:09.636 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.631+0000 7fb93d0bb080 20 remove_watcher() i=5 2020-07-23T12:28:09.641 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.635+0000 7fb93d0bb080 20 remove_watcher() i=6 2020-07-23T12:28:09.643 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.635+0000 7fb93d0bb080 20 remove_watcher() i=7 2020-07-23T12:28:09.650 INFO:teuthology.orchestra.run.smithi137.stdout:[] 2020-07-23T12:28:09.651 DEBUG:tasks.util.rgw: json result: [] 2020-07-23T12:28:09.651 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-23T12:28:09.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-23T12:28:09.652 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-07-23T12:28:09.737 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:09.737 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:09.748 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.743+0000 7f28fc71b080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-23T12:28:09.748 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.743+0000 7f28fc71b080 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-23T12:28:09.760 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.747+0000 7f282f7fe700 20 reqs_thread_entry: start 2020-07-23T12:28:09.761 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.751+0000 7f28fc71b080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.761 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.751+0000 7f28fc71b080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:09.761 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.751+0000 7f28fc71b080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.762 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.751+0000 7f28fc71b080 20 rados_obj.operate() r=0 bl.length=821 2020-07-23T12:28:09.762 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.751+0000 7f28fc71b080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.762 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.751+0000 7f28fc71b080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:09.762 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.751+0000 7f28fc71b080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.762 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.751+0000 7f28fc71b080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:09.763 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.751+0000 7f28fc71b080 20 rados->read ofs=0 len=0 2020-07-23T12:28:09.763 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:09.755+0000 7f28fc71b080 20 rados_obj.operate() r=0 bl.length=358 2020-07-23T12:28:09.769 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"7b7a41ba-6236-454d-8508-c57d66101f1b","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-07-23T12:28:09.769 DEBUG:tasks.util.rgw: json result: {'id': '7b7a41ba-6236-454d-8508-c57d66101f1b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-07-23T12:28:09.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2020-07-23T12:28:09.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2020-07-23T12:28:09.770 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2020-07-23T12:28:09.814 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:09.814 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:09.818 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:09.820+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:11.479 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.480+0000 7f1c016bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:11.479 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.480+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:11.480 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.480+0000 7f1c016bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:11.480 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.480+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:11.481 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.480+0000 7f1c016bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:11.481 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.480+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:11.481 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.484+0000 7f1c016bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:11.499 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.500+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:11.500 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.500+0000 7f1c016bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:11.500 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.500+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:11.501 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.500+0000 7f1c016bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:11.502 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.500+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:11.502 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.504+0000 7f1c016bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:11.502 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.504+0000 7f1c016bb080 10 cannot find current period zonegroup using local zonegroup 2020-07-23T12:28:11.502 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.504+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:11.503 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.504+0000 7f1c016bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:11.503 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.504+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:11.503 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.504+0000 7f1c016bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:11.503 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.504+0000 7f1c016bb080 10 Creating default zonegroup 2020-07-23T12:28:11.503 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.504+0000 7f1c016bb080 20 get_system_obj_state: rctx=0x7ffcbad3c9c0 obj=:.pools.avail state=0x55f242234e40 s->prefetch_data=0 2020-07-23T12:28:11.504 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.504+0000 7f1c016bb080 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-07-23T12:28:11.518 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.520+0000 7f1c016bb080 10 WARNING: store->list_zones() returned r=-2 2020-07-23T12:28:11.518 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.520+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:11.519 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.520+0000 7f1c016bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:11.524 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.524+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:11.525 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.528+0000 7f1c016bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:11.525 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.528+0000 7f1c016bb080 10 could not read realm id: (2) No such file or directory 2020-07-23T12:28:11.525 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.528+0000 7f1c016bb080 10 WARNING: failed to set zone as default, r=-22 2020-07-23T12:28:11.525 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.528+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:11.526 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.528+0000 7f1c016bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:11.530 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.532+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:11.532 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.532+0000 7f1c016bb080 20 rados_obj.operate() r=0 bl.length=805 2020-07-23T12:28:11.533 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.532+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:11.533 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.536+0000 7f1c016bb080 20 rados_obj.operate() r=0 bl.length=358 2020-07-23T12:28:11.533 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.536+0000 7f1c016bb080 20 zonegroup default 2020-07-23T12:28:11.534 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.536+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:11.534 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.536+0000 7f1c016bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:11.535 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.536+0000 7f1c016bb080 10 Cannot find current period zone using local zone 2020-07-23T12:28:11.535 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.536+0000 7f1c016bb080 10 Using default name default 2020-07-23T12:28:11.535 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.536+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:11.536 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.536+0000 7f1c016bb080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:11.536 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.536+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:11.536 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.536+0000 7f1c016bb080 20 rados_obj.operate() r=0 bl.length=805 2020-07-23T12:28:11.536 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.536+0000 7f1c016bb080 20 zone default found 2020-07-23T12:28:11.536 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:11.536+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:13.498 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:13.499+0000 7f1c016bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:13.499 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:13.499+0000 7f1c016bb080 20 rados->read ofs=0 len=0 2020-07-23T12:28:13.500 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:13.499+0000 7f1c016bb080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:13.500 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:13.499+0000 7f1c016bb080 20 started sync module instance, tier type = 2020-07-23T12:28:13.501 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:13.499+0000 7f1c016bb080 20 started zone id=d924b8d5-46ed-40f6-b5cf-405815b8880e (name=default) with tier type = 2020-07-23T12:28:15.543 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:15.542+0000 7f1c016bb080 20 add_watcher() i=0 2020-07-23T12:28:15.544 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:15.542+0000 7f1c016bb080 20 add_watcher() i=1 2020-07-23T12:28:15.544 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:15.542+0000 7f1c016bb080 20 add_watcher() i=2 2020-07-23T12:28:15.544 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:15.542+0000 7f1c016bb080 20 add_watcher() i=3 2020-07-23T12:28:15.544 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:15.542+0000 7f1c016bb080 20 add_watcher() i=4 2020-07-23T12:28:15.544 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:15.542+0000 7f1c016bb080 20 add_watcher() i=5 2020-07-23T12:28:15.545 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:15.542+0000 7f1c016bb080 20 add_watcher() i=6 2020-07-23T12:28:15.548 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:15.546+0000 7f1c016bb080 20 add_watcher() i=7 2020-07-23T12:28:15.549 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:15.546+0000 7f1c016bb080 2 all 8 watchers are set, enabling cache 2020-07-23T12:28:15.549 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:15.546+0000 7f1c016bb080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-23T12:28:15.550 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:15.546+0000 7f1c016bb080 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-23T12:28:15.550 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:15.546+0000 7f1bbe7dc700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-23T12:28:19.509 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.504+0000 7f1c016bb080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-07-23T12:28:19.694 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.688+0000 7f1b34ff9700 20 reqs_thread_entry: start 2020-07-23T12:28:19.694 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.688+0000 7f1c016bb080 20 init_complete bucket index max shards: 11 2020-07-23T12:28:19.694 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.688+0000 7f1b227fc700 20 reqs_thread_entry: start 2020-07-23T12:28:19.695 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.688+0000 7f1b20ff9700 20 reqs_thread_entry: start 2020-07-23T12:28:19.718 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.712+0000 7f1bbe7dc700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-23T12:28:19.721 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.716+0000 7f1c016bb080 20 remove_watcher() i=0 2020-07-23T12:28:19.721 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.716+0000 7f1c016bb080 2 removed watcher, disabling cache 2020-07-23T12:28:19.723 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.720+0000 7f1c016bb080 20 remove_watcher() i=1 2020-07-23T12:28:19.726 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.720+0000 7f1c016bb080 20 remove_watcher() i=2 2020-07-23T12:28:19.729 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.724+0000 7f1c016bb080 20 remove_watcher() i=3 2020-07-23T12:28:19.731 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.724+0000 7f1c016bb080 20 remove_watcher() i=4 2020-07-23T12:28:19.734 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.728+0000 7f1c016bb080 20 remove_watcher() i=5 2020-07-23T12:28:19.737 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.732+0000 7f1c016bb080 20 remove_watcher() i=6 2020-07-23T12:28:19.742 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.736+0000 7f1c016bb080 20 remove_watcher() i=7 2020-07-23T12:28:19.775 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2020-07-23T12:28:19.776 DEBUG:tasks.util.rgw: json result: [] 2020-07-23T12:28:19.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-23T12:28:19.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-23T12:28:19.778 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-07-23T12:28:19.859 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:19.860 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:19.868 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.868+0000 7fb80b03b080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-23T12:28:19.868 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.868+0000 7fb80b03b080 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-23T12:28:19.874 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.872+0000 7fb73f7fe700 20 reqs_thread_entry: start 2020-07-23T12:28:19.875 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.872+0000 7fb80b03b080 20 rados->read ofs=0 len=0 2020-07-23T12:28:19.876 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.876+0000 7fb80b03b080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:19.876 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.876+0000 7fb80b03b080 20 rados->read ofs=0 len=0 2020-07-23T12:28:19.878 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.876+0000 7fb80b03b080 20 rados_obj.operate() r=0 bl.length=805 2020-07-23T12:28:19.878 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.876+0000 7fb80b03b080 20 rados->read ofs=0 len=0 2020-07-23T12:28:19.879 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.876+0000 7fb80b03b080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:19.879 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.876+0000 7fb80b03b080 20 rados->read ofs=0 len=0 2020-07-23T12:28:19.880 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.876+0000 7fb80b03b080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:19.880 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.876+0000 7fb80b03b080 20 rados->read ofs=0 len=0 2020-07-23T12:28:19.881 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.880+0000 7fb80b03b080 20 rados_obj.operate() r=0 bl.length=358 2020-07-23T12:28:19.890 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"d924b8d5-46ed-40f6-b5cf-405815b8880e","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-07-23T12:28:19.891 DEBUG:tasks.util.rgw: json result: {'id': 'd924b8d5-46ed-40f6-b5cf-405815b8880e', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-07-23T12:28:19.892 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2020-07-23T12:28:19.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2020-07-23T12:28:19.895 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2020-07-23T12:28:19.966 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:19.967 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:19.979 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.976+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:19.980 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.980+0000 7ff8cd9ef080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:19.980 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.980+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:19.982 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.980+0000 7ff8cd9ef080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:19.982 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.980+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:19.983 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.980+0000 7ff8cd9ef080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:19.983 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.980+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:19.983 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:19.980+0000 7ff8cd9ef080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:20.007 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.004+0000 7ff8cd9ef080 20 RGWRados::pool_iterate: got zone_info.d924b8d5-46ed-40f6-b5cf-405815b8880e 2020-07-23T12:28:20.008 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.004+0000 7ff8cd9ef080 20 RGWRados::pool_iterate: got zonegroup_info.fb786df1-29ed-495b-aff3-159b9a394255 2020-07-23T12:28:20.008 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.004+0000 7ff8cd9ef080 20 RGWRados::pool_iterate: got zone_names.default 2020-07-23T12:28:20.008 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.004+0000 7ff8cd9ef080 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-07-23T12:28:20.009 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.004+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.009 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.004+0000 7ff8cd9ef080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:20.009 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.004+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.011 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.008+0000 7ff8cd9ef080 20 rados_obj.operate() r=0 bl.length=821 2020-07-23T12:28:20.011 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.008+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.011 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.008+0000 7ff8cd9ef080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:20.011 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.008+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.012 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.008+0000 7ff8cd9ef080 20 rados_obj.operate() r=0 bl.length=358 2020-07-23T12:28:20.012 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.008+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.012 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.008+0000 7ff8cd9ef080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:20.013 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.008+0000 7ff8cd9ef080 10 cannot find current period zonegroup using local zonegroup 2020-07-23T12:28:20.013 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.008+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.014 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.008+0000 7ff8cd9ef080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:20.015 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.008+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.015 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.012+0000 7ff8cd9ef080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:20.015 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.012+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.016 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.012+0000 7ff8cd9ef080 20 rados_obj.operate() r=0 bl.length=358 2020-07-23T12:28:20.016 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.012+0000 7ff8cd9ef080 20 zonegroup default 2020-07-23T12:28:20.016 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.012+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.017 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.012+0000 7ff8cd9ef080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:20.017 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.012+0000 7ff8cd9ef080 10 Cannot find current period zone using local zone 2020-07-23T12:28:20.017 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.012+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.018 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.012+0000 7ff8cd9ef080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:20.018 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.012+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.018 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.012+0000 7ff8cd9ef080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:20.019 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.012+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.020 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.012+0000 7ff8cd9ef080 20 rados_obj.operate() r=0 bl.length=821 2020-07-23T12:28:20.020 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.012+0000 7ff8cd9ef080 20 zone default found 2020-07-23T12:28:20.021 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.012+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.021 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.016+0000 7ff8cd9ef080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:20.022 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.016+0000 7ff8cd9ef080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.022 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.016+0000 7ff8cd9ef080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:20.022 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.016+0000 7ff8cd9ef080 20 started sync module instance, tier type = 2020-07-23T12:28:20.023 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.016+0000 7ff8cd9ef080 20 started zone id=d924b8d5-46ed-40f6-b5cf-405815b8880e (name=default) with tier type = 2020-07-23T12:28:20.051 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.048+0000 7ff8cd9ef080 20 add_watcher() i=0 2020-07-23T12:28:20.052 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.048+0000 7ff8cd9ef080 20 add_watcher() i=1 2020-07-23T12:28:20.052 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.048+0000 7ff8cd9ef080 20 add_watcher() i=2 2020-07-23T12:28:20.052 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.048+0000 7ff8cd9ef080 20 add_watcher() i=3 2020-07-23T12:28:20.053 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.048+0000 7ff8cd9ef080 20 add_watcher() i=4 2020-07-23T12:28:20.053 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.048+0000 7ff8cd9ef080 20 add_watcher() i=5 2020-07-23T12:28:20.054 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.048+0000 7ff8cd9ef080 20 add_watcher() i=6 2020-07-23T12:28:20.054 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.052+0000 7ff8cd9ef080 20 add_watcher() i=7 2020-07-23T12:28:20.055 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.052+0000 7ff8cd9ef080 2 all 8 watchers are set, enabling cache 2020-07-23T12:28:20.055 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.052+0000 7ff8cd9ef080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-23T12:28:20.055 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.052+0000 7ff8cd9ef080 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-23T12:28:20.056 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.052+0000 7ff88afdd700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-23T12:28:20.150 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.148+0000 7ff7f97fa700 20 reqs_thread_entry: start 2020-07-23T12:28:20.151 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.148+0000 7ff7f77f6700 20 reqs_thread_entry: start 2020-07-23T12:28:20.151 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.148+0000 7ff8cd9ef080 20 init_complete bucket index max shards: 11 2020-07-23T12:28:20.151 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.148+0000 7ff7f5ff3700 20 reqs_thread_entry: start 2020-07-23T12:28:20.179 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.172+0000 7ff88afdd700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-07-23T12:28:20.181 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.176+0000 7ff8cd9ef080 20 remove_watcher() i=0 2020-07-23T12:28:20.181 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.176+0000 7ff8cd9ef080 2 removed watcher, disabling cache 2020-07-23T12:28:20.181 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.180+0000 7ff8cd9ef080 20 remove_watcher() i=1 2020-07-23T12:28:20.183 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.180+0000 7ff8cd9ef080 20 remove_watcher() i=2 2020-07-23T12:28:20.186 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.184+0000 7ff8cd9ef080 20 remove_watcher() i=3 2020-07-23T12:28:20.189 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.188+0000 7ff8cd9ef080 20 remove_watcher() i=4 2020-07-23T12:28:20.192 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.192+0000 7ff8cd9ef080 20 remove_watcher() i=5 2020-07-23T12:28:20.194 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.192+0000 7ff8cd9ef080 20 remove_watcher() i=6 2020-07-23T12:28:20.197 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.196+0000 7ff8cd9ef080 20 remove_watcher() i=7 2020-07-23T12:28:20.205 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2020-07-23T12:28:20.205 DEBUG:tasks.util.rgw: json result: [] 2020-07-23T12:28:20.206 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-23T12:28:20.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-07-23T12:28:20.207 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-07-23T12:28:20.288 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:20.288 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:20.299 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.296+0000 7fa3b4411080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-07-23T12:28:20.300 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.296+0000 7fa3b4411080 20 check_secure_mon_conn(): mode 1 is insecure 2020-07-23T12:28:20.305 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.304+0000 7fa2e7fff700 20 reqs_thread_entry: start 2020-07-23T12:28:20.306 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.304+0000 7fa3b4411080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.306 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.304+0000 7fa3b4411080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:20.307 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.304+0000 7fa3b4411080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.308 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.308+0000 7fa3b4411080 20 rados_obj.operate() r=0 bl.length=821 2020-07-23T12:28:20.308 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.308+0000 7fa3b4411080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.309 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.308+0000 7fa3b4411080 20 rados_obj.operate() r=-2 bl.length=0 2020-07-23T12:28:20.310 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.308+0000 7fa3b4411080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.310 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.308+0000 7fa3b4411080 20 rados_obj.operate() r=0 bl.length=46 2020-07-23T12:28:20.310 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.308+0000 7fa3b4411080 20 rados->read ofs=0 len=0 2020-07-23T12:28:20.310 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:20.308+0000 7fa3b4411080 20 rados_obj.operate() r=0 bl.length=358 2020-07-23T12:28:20.318 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"d924b8d5-46ed-40f6-b5cf-405815b8880e","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-07-23T12:28:20.318 DEBUG:tasks.util.rgw: json result: {'id': 'd924b8d5-46ed-40f6-b5cf-405815b8880e', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-07-23T12:28:20.319 INFO:tasks.rgw:Starting rgw... 2020-07-23T12:28:20.319 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2020-07-23T12:28:20.320 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-07-23T12:28:20.320 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-07-23T12:28:20.321 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-07-23T12:28:20.324 INFO:tasks.rgw.c1.client.0:Started 2020-07-23T12:28:20.324 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2020-07-23T12:28:20.324 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-07-23T12:28:20.325 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-07-23T12:28:20.325 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-07-23T12:28:20.328 INFO:tasks.rgw.c1.client.1:Started 2020-07-23T12:28:20.328 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2020-07-23T12:28:20.328 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-07-23T12:28:20.329 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-07-23T12:28:20.330 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-07-23T12:28:20.361 INFO:tasks.rgw.c2.client.0:Started 2020-07-23T12:28:20.362 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2020-07-23T12:28:20.362 INFO:tasks.rgw:Using civetweb as radosgw frontend 2020-07-23T12:28:20.362 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-07-23T12:28:20.363 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-07-23T12:28:20.365 INFO:tasks.rgw.c2.client.1:Started 2020-07-23T12:28:20.365 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi137.front.sepia.ceph.com:8000/ 2020-07-23T12:28:20.366 INFO:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8000/ 2020-07-23T12:28:20.406 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-23T12:28:20.406 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2020-07-23T12:28:20.408 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi137.front.sepia.ceph.com port 8000: Connection refused 2020-07-23T12:28:20.409 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-23T12:28:21.411 INFO:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8000/ 2020-07-23T12:28:21.427 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-23T12:28:21.427 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2020-07-23T12:28:21.430 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2020-07-23T12:28:21.431 INFO:teuthology.orchestra.run.smithi137.stdout:anonymous 2020-07-23T12:28:21.431 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi137.front.sepia.ceph.com:8001/ 2020-07-23T12:28:21.432 INFO:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8001/ 2020-07-23T12:28:21.485 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-23T12:28:21.486 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2020-07-23T12:28:21.490 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2020-07-23T12:28:21.491 INFO:teuthology.orchestra.run.smithi137.stdout:anonymous 2020-07-23T12:28:21.491 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi133.front.sepia.ceph.com:8000/ 2020-07-23T12:28:21.492 INFO:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8000/ 2020-07-23T12:28:21.504 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-23T12:28:21.505 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2020-07-23T12:28:21.509 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-07-23T12:28:21.510 INFO:teuthology.orchestra.run.smithi133.stdout:anonymous 2020-07-23T12:28:21.511 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi133.front.sepia.ceph.com:8001/ 2020-07-23T12:28:21.511 INFO:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8001/ 2020-07-23T12:28:21.566 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-23T12:28:21.567 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2020-07-23T12:28:21.571 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2020-07-23T12:28:21.572 INFO:teuthology.orchestra.run.smithi133.stdout:anonymous 2020-07-23T12:28:21.572 INFO:teuthology.run_tasks:Running task rgw-multisite... 2020-07-23T12:28:22.030 INFO:tasks.rgw_multisite:creating realm.. 2020-07-23T12:28:22.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:22.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:22.032 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:28:22.069 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:22.069 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:22.118 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","name":"test-realm","current_period":"c2d2f17c-d146-4cb4-9071-2453df8a0a1f","epoch":1} 2020-07-23T12:28:22.119 INFO:tasks.rgw_multisite:creating master zonegroup.. 2020-07-23T12:28:22.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:22.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:22.120 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:28:22.199 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:22.200 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:22.221 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:22.215+0000 7f564e326080 -1 old format 2020-07-23T12:28:22.246 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}} 2020-07-23T12:28:22.247 INFO:tasks.rgw_multisite:creating master zone.. 2020-07-23T12:28:22.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--endpoints', 'http://smithi137.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:22.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--endpoints', 'http://smithi137.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:22.248 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key T855BRGPSH14I4Y2 --secret mMgfonT3dSbKAOArpyZfLcrPB3lPybUX --endpoints http://smithi137.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:28:22.335 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:22.336 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:22.403 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"T855BRGPSH14I4Y2","secret_key":"mMgfonT3dSbKAOArpyZfLcrPB3lPybUX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca"} 2020-07-23T12:28:22.404 INFO:teuthology.orchestra.run.smithi137:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2020-07-23T12:28:22.936 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'test-zone1.rgw.buckets.data' created 2020-07-23T12:28:22.956 INFO:teuthology.orchestra.run.smithi137:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2020-07-23T12:28:23.950 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2020-07-23T12:28:23.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:23.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:23.972 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:28:24.018 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:24.018 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:24.042 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"T855BRGPSH14I4Y2","secret_key":"mMgfonT3dSbKAOArpyZfLcrPB3lPybUX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-07-23T12:28:24.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:24.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:24.044 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:28:24.124 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:24.124 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:32.198 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","epoch":1,"predecessor_uuid":"c2d2f17c-d146-4cb4-9071-2453df8a0a1f","sync_status":[],"period_map":{"id":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","zones":[{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0378667f-02f5-4ce6-824e-e2a396c12fc3","val":68328169}]},"master_zonegroup":"test-zonegroup","master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","realm_name":"test-realm","realm_epoch":2} 2020-07-23T12:28:32.199 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2020-07-23T12:28:32.200 INFO:teuthology.orchestra.run:waiting for 300 2020-07-23T12:28:32.243 INFO:tasks.rgw.c1.client.0.smithi137.stdout:2020-07-23T12:28:32.239+0000 7f6139cac700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 13373) UID: 0 2020-07-23T12:28:32.244 INFO:tasks.rgw.c1.client.0.smithi137.stdout:2020-07-23T12:28:32.239+0000 7f61512300c0 -1 shutting down 2020-07-23T12:28:36.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2020-07-23T12:28:38.302 INFO:tasks.rgw.c1.client.0:Stopped 2020-07-23T12:28:38.303 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-07-23T12:28:38.303 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-07-23T12:28:38.309 INFO:tasks.rgw.c1.client.0:Started 2020-07-23T12:28:38.310 INFO:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8000 2020-07-23T12:28:38.328 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-23T12:28:38.328 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2020-07-23T12:28:38.332 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi137.front.sepia.ceph.com port 8000: Connection refused 2020-07-23T12:28:38.333 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-23T12:28:39.338 INFO:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8000 2020-07-23T12:28:39.353 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-23T12:28:39.354 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2020-07-23T12:28:39.358 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2020-07-23T12:28:39.358 INFO:teuthology.orchestra.run.smithi137.stdout:anonymous 2020-07-23T12:28:39.359 INFO:tasks.rgw_multisite:creating admin user.. 2020-07-23T12:28:39.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:39.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:39.360 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key T855BRGPSH14I4Y2 --secret mMgfonT3dSbKAOArpyZfLcrPB3lPybUX --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:28:39.449 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:39.449 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:39.645 INFO:teuthology.orchestra.run.smithi137.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"T855BRGPSH14I4Y2","secret_key":"mMgfonT3dSbKAOArpyZfLcrPB3lPybUX"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":"true","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2020-07-23T12:28:39.666 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-07-23T12:28:39.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi137.front.sepia.ceph.com:8000', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:39.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi137.front.sepia.ceph.com:8000', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:39.669 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi137.front.sepia.ceph.com:8000 --access-key T855BRGPSH14I4Y2 --secret mMgfonT3dSbKAOArpyZfLcrPB3lPybUX --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:28:39.714 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:39.714 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:39.838 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","name":"test-realm","current_period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","epoch":2} 2020-07-23T12:28:39.839 INFO:tasks.rgw_multisite:creating zone.. 2020-07-23T12:28:39.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--endpoints', 'http://smithi133.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:39.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--endpoints', 'http://smithi133.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:39.840 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key T855BRGPSH14I4Y2 --secret mMgfonT3dSbKAOArpyZfLcrPB3lPybUX --endpoints http://smithi133.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:28:39.927 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:39.928 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:39.991 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:39.988+0000 7f395d3da080 0 failed reading obj info from .rgw.root:zone_info.0378667f-02f5-4ce6-824e-e2a396c12fc3: (2) No such file or directory 2020-07-23T12:28:39.991 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:28:39.988+0000 7f395d3da080 0 WARNING: could not read zone params for zone id=0378667f-02f5-4ce6-824e-e2a396c12fc3 name=test-zone1 2020-07-23T12:28:40.004 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"T855BRGPSH14I4Y2","secret_key":"mMgfonT3dSbKAOArpyZfLcrPB3lPybUX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca"} 2020-07-23T12:28:40.005 INFO:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2020-07-23T12:28:40.434 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'test-zone2.rgw.buckets.data' created 2020-07-23T12:28:40.452 INFO:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2020-07-23T12:28:41.474 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2020-07-23T12:28:41.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:41.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:41.490 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:28:41.530 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:41.530 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:41.557 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"T855BRGPSH14I4Y2","secret_key":"mMgfonT3dSbKAOArpyZfLcrPB3lPybUX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-07-23T12:28:41.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:41.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:41.560 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:28:41.637 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:41.637 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:42.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2020-07-23T12:28:49.647 INFO:teuthology.orchestra.run.smithi133.stderr:Sending period to new master zone 0378667f-02f5-4ce6-824e-e2a396c12fc3 2020-07-23T12:28:50.752 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","epoch":2,"predecessor_uuid":"c2d2f17c-d146-4cb4-9071-2453df8a0a1f","sync_status":[],"period_map":{"id":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","zones":[{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0378667f-02f5-4ce6-824e-e2a396c12fc3","val":68328169},{"key":"42997773-3698-4448-bf7b-a00a5846d584","val":1484105665}]},"master_zonegroup":"test-zonegroup","master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","realm_name":"test-realm","realm_epoch":2} 2020-07-23T12:28:50.753 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2020-07-23T12:28:50.754 INFO:teuthology.orchestra.run:waiting for 300 2020-07-23T12:28:50.796 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:28:50.796+0000 7f3334e09700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 13258) UID: 0 2020-07-23T12:28:50.796 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:28:50.796+0000 7f334c38d0c0 -1 shutting down 2020-07-23T12:28:56.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2020-07-23T12:28:56.857 INFO:tasks.rgw.c2.client.0:Stopped 2020-07-23T12:28:56.858 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-07-23T12:28:56.858 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-07-23T12:28:56.862 INFO:tasks.rgw.c2.client.0:Started 2020-07-23T12:28:56.862 INFO:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8000 2020-07-23T12:28:56.880 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-23T12:28:56.881 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2020-07-23T12:28:56.886 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi133.front.sepia.ceph.com port 8000: Connection refused 2020-07-23T12:28:56.887 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-23T12:28:57.888 INFO:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8000 2020-07-23T12:28:57.913 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-23T12:28:57.913 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2020-07-23T12:28:57.917 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2020-07-23T12:28:57.918 INFO:teuthology.orchestra.run.smithi133.stdout:anonymous 2020-07-23T12:28:57.919 INFO:tasks.rgw_multisite:creating zone.. 2020-07-23T12:28:57.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--endpoints', 'http://smithi137.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:57.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--endpoints', 'http://smithi137.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:28:57.921 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key T855BRGPSH14I4Y2 --secret mMgfonT3dSbKAOArpyZfLcrPB3lPybUX --endpoints http://smithi137.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:28:57.978 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:28:57.978 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:28:58.023 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:58.019+0000 7f8d81c36080 0 failed reading obj info from .rgw.root:zone_info.42997773-3698-4448-bf7b-a00a5846d584: (2) No such file or directory 2020-07-23T12:28:58.023 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:28:58.019+0000 7f8d81c36080 0 WARNING: could not read zone params for zone id=42997773-3698-4448-bf7b-a00a5846d584 name=test-zone2 2020-07-23T12:28:58.032 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"41767e64-47c7-45a2-9984-76631257f2f6","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"T855BRGPSH14I4Y2","secret_key":"mMgfonT3dSbKAOArpyZfLcrPB3lPybUX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca"} 2020-07-23T12:28:58.033 INFO:teuthology.orchestra.run.smithi137:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2020-07-23T12:28:58.982 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'test-zone3.rgw.buckets.data' created 2020-07-23T12:28:59.001 INFO:teuthology.orchestra.run.smithi137:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2020-07-23T12:28:59.983 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2020-07-23T12:29:00.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:29:00.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:29:00.004 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:29:00.059 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:29:00.060 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:29:00.083 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"41767e64-47c7-45a2-9984-76631257f2f6","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"T855BRGPSH14I4Y2","secret_key":"mMgfonT3dSbKAOArpyZfLcrPB3lPybUX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-07-23T12:29:00.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:29:00.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:29:00.087 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:29:00.125 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:29:00.125 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:29:02.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2020-07-23T12:29:08.199 INFO:teuthology.orchestra.run.smithi137.stderr:Sending period to new master zone 0378667f-02f5-4ce6-824e-e2a396c12fc3 2020-07-23T12:29:09.294 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","epoch":3,"predecessor_uuid":"c2d2f17c-d146-4cb4-9071-2453df8a0a1f","sync_status":[],"period_map":{"id":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","zones":[{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"41767e64-47c7-45a2-9984-76631257f2f6","name":"test-zone3","endpoints":["http://smithi137.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0378667f-02f5-4ce6-824e-e2a396c12fc3","val":68328169},{"key":"41767e64-47c7-45a2-9984-76631257f2f6","val":2499065231},{"key":"42997773-3698-4448-bf7b-a00a5846d584","val":1484105665}]},"master_zonegroup":"test-zonegroup","master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","realm_name":"test-realm","realm_epoch":2} 2020-07-23T12:29:09.295 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-07-23T12:29:09.296 INFO:teuthology.orchestra.run:waiting for 300 2020-07-23T12:29:09.342 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:29:09.339+0000 7f8f7f7fe700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 13377) UID: 0 2020-07-23T12:29:09.343 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:29:09.339+0000 7f8fa30150c0 -1 shutting down 2020-07-23T12:29:11.095 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-07-23T12:29:15.398 INFO:tasks.rgw.c1.client.1:Stopped 2020-07-23T12:29:15.399 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-07-23T12:29:15.399 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-07-23T12:29:15.404 INFO:tasks.rgw.c1.client.1:Started 2020-07-23T12:29:15.405 INFO:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8001 2020-07-23T12:29:15.423 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-23T12:29:15.423 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2020-07-23T12:29:15.427 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi137.front.sepia.ceph.com port 8001: Connection refused 2020-07-23T12:29:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-23T12:29:16.429 INFO:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8001 2020-07-23T12:29:16.444 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-23T12:29:16.444 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2020-07-23T12:29:16.448 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2020-07-23T12:29:16.449 INFO:teuthology.orchestra.run.smithi137.stdout:anonymous 2020-07-23T12:29:16.451 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-07-23T12:29:16.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi137.front.sepia.ceph.com:8000', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--url', 'http://smithi137.front.sepia.ceph.com:8000', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:29:16.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi137.front.sepia.ceph.com:8000', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--url', 'http://smithi137.front.sepia.ceph.com:8000', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:29:16.455 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi137.front.sepia.ceph.com:8000 --access-key T855BRGPSH14I4Y2 --secret mMgfonT3dSbKAOArpyZfLcrPB3lPybUX --url http://smithi137.front.sepia.ceph.com:8000 --access-key T855BRGPSH14I4Y2 --secret mMgfonT3dSbKAOArpyZfLcrPB3lPybUX --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:29:16.503 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:29:16.504 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:29:16.546 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:29:16.544+0000 7eff99e03700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.bca7b68d-77da-418f-a5f9-ecb5e2e483ca.control err=-107 2020-07-23T12:29:16.572 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","name":"test-realm","current_period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","epoch":2} 2020-07-23T12:29:16.572 INFO:tasks.rgw_multisite:creating zone.. 2020-07-23T12:29:16.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--endpoints', 'http://smithi133.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:29:16.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--endpoints', 'http://smithi133.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:29:16.576 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key T855BRGPSH14I4Y2 --secret mMgfonT3dSbKAOArpyZfLcrPB3lPybUX --endpoints http://smithi133.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:29:16.666 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:29:16.666 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:29:16.730 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:29:16.728+0000 7fa5d8ee9080 0 failed reading obj info from .rgw.root:zone_info.0378667f-02f5-4ce6-824e-e2a396c12fc3: (2) No such file or directory 2020-07-23T12:29:16.730 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:29:16.728+0000 7fa5d8ee9080 0 WARNING: could not read zone params for zone id=0378667f-02f5-4ce6-824e-e2a396c12fc3 name=test-zone1 2020-07-23T12:29:16.732 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:29:16.732+0000 7fa5d8ee9080 0 failed reading obj info from .rgw.root:zone_info.41767e64-47c7-45a2-9984-76631257f2f6: (2) No such file or directory 2020-07-23T12:29:16.732 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:29:16.732+0000 7fa5d8ee9080 0 WARNING: could not read zone params for zone id=41767e64-47c7-45a2-9984-76631257f2f6 name=test-zone3 2020-07-23T12:29:16.743 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"32a3ec76-95f5-4863-891c-667a67c89f48","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"T855BRGPSH14I4Y2","secret_key":"mMgfonT3dSbKAOArpyZfLcrPB3lPybUX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca"} 2020-07-23T12:29:16.743 INFO:teuthology.orchestra.run.smithi133:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2020-07-23T12:29:16.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-07-23T12:29:17.476 INFO:teuthology.orchestra.run.smithi133.stderr:pool 'test-zone4.rgw.buckets.data' created 2020-07-23T12:29:17.491 INFO:teuthology.orchestra.run.smithi133:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2020-07-23T12:29:18.489 INFO:teuthology.orchestra.run.smithi133.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2020-07-23T12:29:18.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:29:18.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:29:18.521 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:29:18.574 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:29:18.574 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:29:18.599 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"32a3ec76-95f5-4863-891c-667a67c89f48","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"T855BRGPSH14I4Y2","secret_key":"mMgfonT3dSbKAOArpyZfLcrPB3lPybUX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":""} 2020-07-23T12:29:18.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:29:18.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:29:18.601 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:29:18.680 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:29:18.681 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:29:26.703 INFO:teuthology.orchestra.run.smithi133.stderr:Sending period to new master zone 0378667f-02f5-4ce6-824e-e2a396c12fc3 2020-07-23T12:29:27.850 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","epoch":4,"predecessor_uuid":"c2d2f17c-d146-4cb4-9071-2453df8a0a1f","sync_status":[],"period_map":{"id":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","zones":[{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"32a3ec76-95f5-4863-891c-667a67c89f48","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"41767e64-47c7-45a2-9984-76631257f2f6","name":"test-zone3","endpoints":["http://smithi137.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0378667f-02f5-4ce6-824e-e2a396c12fc3","val":68328169},{"key":"32a3ec76-95f5-4863-891c-667a67c89f48","val":3058828581},{"key":"41767e64-47c7-45a2-9984-76631257f2f6","val":2499065231},{"key":"42997773-3698-4448-bf7b-a00a5846d584","val":1484105665}]},"master_zonegroup":"test-zonegroup","master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","realm_name":"test-realm","realm_epoch":2} 2020-07-23T12:29:27.851 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2020-07-23T12:29:27.852 INFO:teuthology.orchestra.run:waiting for 300 2020-07-23T12:29:27.895 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:29:27.896+0000 7fb9ba577700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 13261) UID: 0 2020-07-23T12:29:27.896 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:29:27.896+0000 7fb9d1afb0c0 -1 shutting down 2020-07-23T12:29:30.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2020-07-23T12:29:33.959 INFO:tasks.rgw.c2.client.1:Stopped 2020-07-23T12:29:33.960 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-07-23T12:29:33.961 INFO:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-07-23T12:29:33.964 INFO:tasks.rgw.c2.client.1:Started 2020-07-23T12:29:33.964 INFO:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8001 2020-07-23T12:29:33.982 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-23T12:29:33.982 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2020-07-23T12:29:33.986 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi133.front.sepia.ceph.com port 8001: Connection refused 2020-07-23T12:29:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-23T12:29:34.988 INFO:teuthology.orchestra.run.smithi133:> curl http://smithi133.front.sepia.ceph.com:8001 2020-07-23T12:29:35.002 INFO:teuthology.orchestra.run.smithi133.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-23T12:29:35.002 INFO:teuthology.orchestra.run.smithi133.stderr: Dload Upload Total Spent Left Speed 2020-07-23T12:29:35.007 INFO:teuthology.orchestra.run.smithi133.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 33500 0 --:--:-- --:--:-- --:--:-- 33500 2020-07-23T12:29:35.007 INFO:teuthology.orchestra.run.smithi133.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-005f1982af-115d-test-zone4","HostId":"115d-test-zone4-test-zonegroup"} 2020-07-23T12:29:35.008 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2020-07-23T12:29:35.008 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2020-07-23T12:29:35.095 INFO:tasks.rgw_multisite_tests:creating test user.. 2020-07-23T12:29:35.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:29:35.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:29:35.097 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:29:35.143 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:29:35.143 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:29:35.343 INFO:teuthology.orchestra.run.smithi137.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"3LB700X01JPXCK7PUDSL","secret_key":"c2is7Bj7MU9BGDg8JL2j3PSjevW4tNGKRSa122eo"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2020-07-23T12:29:35.369 INFO:rgw_multi.tests:meta checkpoint 2020-07-23T12:29:35.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:29:35.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:29:35.371 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:29:35.408 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:29:35.409 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:29:35.604 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:29:35.638 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-23T12:29:35.639 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:29:35.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:29:35.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:29:35.640 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:29:35.681 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:29:35.682 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:29:35.905 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:29:35.906 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:29:36.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2020-07-23T12:29:40.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:29:40.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:29:40.909 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:29:40.950 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:29:40.950 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:29:41.178 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:29:41.179 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:29:46.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:29:46.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:29:46.184 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:29:46.231 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:29:46.231 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:29:46.437 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:29:46.438 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:29:51.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:29:51.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:29:51.442 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:29:51.514 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:29:51.515 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:29:51.715 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:29:51.716 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:29:56.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:29:56.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:29:56.719 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:29:56.760 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:29:56.760 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:29:56.991 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:29:56.992 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:30:01.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:01.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:01.996 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:30:02.037 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:30:02.037 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:30:02.264 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:30:02.265 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:30:07.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:07.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:07.268 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:30:07.315 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:30:07.315 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:30:07.530 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:30:07.530 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:30:12.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:12.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:12.533 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:30:12.576 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:30:12.577 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:30:12.785 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:30:12.786 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:30:17.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:17.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:17.788 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:30:17.838 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:30:17.839 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:30:18.062 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:30:18.063 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:30:23.064 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:23.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:23.069 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:30:23.118 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:30:23.118 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:30:23.343 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:30:23.343 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:30:28.345 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:28.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:28.346 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:30:28.395 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:30:28.395 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:30:28.614 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:30:28.614 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:30:33.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:33.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:33.618 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:30:33.670 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:30:33.671 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:30:33.902 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:30:33.903 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:30:38.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:38.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:38.907 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:30:38.947 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:30:38.948 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:30:39.166 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:30:39.167 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:30:44.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:44.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:44.184 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:30:44.230 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:30:44.231 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:30:44.458 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:30:44.459 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:30:49.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:49.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:49.463 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:30:49.507 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:30:49.508 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:30:49.730 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:30:49.731 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:30:54.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:54.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:30:54.735 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:30:54.777 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:30:54.778 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:30:55.008 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:30:55.009 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:31:00.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:00.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:00.012 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:00.058 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:00.058 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:00.257 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:00.258 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:31:00.258 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:31:00.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:00.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:00.260 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:00.306 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:00.307 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:00.518 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:00.519 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:31:05.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:05.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:05.524 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:05.566 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:05.567 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:05.805 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:05.806 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:31:10.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:10.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:10.812 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:10.852 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:10.853 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:11.087 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:11.087 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:31:16.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:16.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:16.091 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:16.134 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:16.135 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:16.354 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:16.355 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:31:21.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:21.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:21.578 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:21.621 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:21.621 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:21.847 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:21.848 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:31:26.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:26.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:26.863 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:26.925 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:26.926 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:27.145 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:27.146 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1595507375.329399_72.1 target= 2020-07-23T12:31:32.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:32.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:32.149 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:32.191 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:32.192 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:32.414 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:32.416 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:31:32.416 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:31:32.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:32.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:32.418 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:32.458 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:32.458 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:32.731 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:32.732 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:31:32.733 INFO:rgw_multi.tests:meta checkpoint 2020-07-23T12:31:32.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:32.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:32.735 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:32.775 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:32.775 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:32.979 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:31:33.007 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-07-23T12:31:33.008 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:31:33.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:33.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:33.009 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:33.051 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:33.051 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:33.263 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:33.264 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:31:33.265 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:31:33.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:33.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:33.266 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:33.310 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:33.310 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:33.540 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:33.541 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:31:33.541 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:31:33.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:33.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:33.543 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:33.593 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:33.594 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:33.857 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:33.857 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:31:33.858 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2020-07-23T12:31:33.881 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-1 2020-07-23T12:31:37.068 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-2 2020-07-23T12:31:40.550 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-3 2020-07-23T12:31:40.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:40.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:40.629 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:40.664 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:40.665 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:40.834 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:31:40.871 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595507500.607977_91.1","last_update":"2020-07-23T12:31:40.607977Z"},{"marker":"","last_update":"0.000000"}] 2020-07-23T12:31:40.872 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:31:40.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:40.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:40.883 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:40.937 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:40.938 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:41.170 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507497.110118_87.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:31:37.110118Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:41.171 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1595507500.607977_91.1 target=1_1595507497.110118_87.1 2020-07-23T12:31:46.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:46.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:46.175 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:46.220 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:46.220 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:46.442 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507500.607977_91.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:31:40.607977Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:46.443 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:31:46.443 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:31:46.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:46.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:46.445 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:46.491 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:46.491 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:46.724 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507500.607977_91.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:31:40.607977Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:46.725 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:31:46.726 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:31:46.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:46.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:46.728 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:46.769 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:46.770 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:47.034 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507500.607977_91.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:31:40.607977Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:47.035 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:31:48.290 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2020-07-23T12:31:48.293 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-4 2020-07-23T12:31:48.321 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-5 2020-07-23T12:31:48.378 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-6 2020-07-23T12:31:48.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:48.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:48.427 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:48.467 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:48.467 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:48.657 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:31:48.690 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595507508.407314_105.1","last_update":"2020-07-23T12:31:48.407314Z"},{"marker":"","last_update":"0.000000"}] 2020-07-23T12:31:48.691 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:31:48.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:48.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:48.694 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:48.734 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:48.734 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:48.973 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507500.607977_91.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:31:40.607977Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:48.974 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1595507508.407314_105.1 target=1_1595507500.607977_91.1 2020-07-23T12:31:53.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:53.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:53.979 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:54.030 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:54.031 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:54.259 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507508.407314_105.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:31:48.407314Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:54.260 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:31:54.261 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:31:54.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:54.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:54.262 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:54.304 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:54.305 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:54.528 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507508.407314_105.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:31:48.407314Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:54.529 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:31:54.530 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:31:54.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:54.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:54.535 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:54.576 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:54.577 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:54.840 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507508.407314_105.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:31:48.407314Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:54.841 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:31:56.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:56.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:56.639 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:56.674 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:56.674 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:56.875 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:31:56.910 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595507516.493752_143.1","last_update":"2020-07-23T12:31:56.493752Z"},{"marker":"","last_update":"0.000000"}] 2020-07-23T12:31:56.911 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:31:56.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:56.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:56.914 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:56.955 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:56.956 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:57.178 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507516.493752_143.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:31:56.493752Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:57.179 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:31:57.180 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:31:57.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:57.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:57.184 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:57.224 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:57.224 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:57.472 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507516.493752_143.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:31:56.493752Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:57.473 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:31:57.474 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:31:57.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:57.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:57.475 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:57.519 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:57.520 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:57.779 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507516.493752_143.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:31:56.493752Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:57.780 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:31:59.181 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2020-07-23T12:31:59.183 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-7 2020-07-23T12:31:59.214 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-8 2020-07-23T12:31:59.280 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-9 2020-07-23T12:31:59.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:59.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:59.340 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:59.375 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:59.376 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:59.574 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:31:59.607 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595507519.320666_157.1","last_update":"2020-07-23T12:31:59.320666Z"},{"marker":"","last_update":"0.000000"}] 2020-07-23T12:31:59.608 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:31:59.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:59.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:31:59.611 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:31:59.650 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:31:59.650 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:31:59.868 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507516.493752_143.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:31:56.493752Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:31:59.869 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1595507519.320666_157.1 target=1_1595507516.493752_143.1 2020-07-23T12:32:04.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:04.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:04.873 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:04.913 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:04.914 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:05.152 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507519.320666_157.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:31:59.320666Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:32:05.152 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:32:05.153 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:32:05.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:05.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:05.155 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:05.197 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:05.197 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:05.438 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507519.320666_157.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:31:59.320666Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:32:05.439 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:32:05.440 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:32:05.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:05.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:05.442 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:05.485 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:05.485 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:05.753 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507519.320666_157.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:31:59.320666Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:32:05.754 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:32:08.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:08.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:08.203 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:08.237 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:08.238 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:08.425 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:32:08.466 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595507528.176421_165.1","last_update":"2020-07-23T12:32:08.176421Z"},{"marker":"","last_update":"0.000000"}] 2020-07-23T12:32:08.467 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:32:08.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:08.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:08.468 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:08.509 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:08.510 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:08.738 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507519.320666_157.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:31:59.320666Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:32:08.739 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1595507528.176421_165.1 target=1_1595507519.320666_157.1 2020-07-23T12:32:13.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:13.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:13.758 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:13.806 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:13.806 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:14.047 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:32:14.047 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:32:14.048 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:32:14.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:14.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:14.050 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:14.090 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:14.090 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:14.334 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:32:14.335 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:32:14.336 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:32:14.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:14.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:14.339 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:14.382 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:14.383 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:14.656 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:32:14.657 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:32:16.660 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2020-07-23T12:32:16.661 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-10 2020-07-23T12:32:17.185 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-11 2020-07-23T12:32:18.152 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-12 2020-07-23T12:32:20.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:20.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:20.661 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:20.701 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:20.702 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:20.898 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:32:20.928 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595507528.176421_165.1","last_update":"2020-07-23T12:32:08.176421Z"},{"marker":"1_1595507538.853263_65.1","last_update":"2020-07-23T12:32:18.853263Z"}] 2020-07-23T12:32:20.928 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:32:20.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:20.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:20.930 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:20.972 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:21.236 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:21.237 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507538.853263_65.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:18.853263Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:32:21.238 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:32:21.238 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:32:21.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:21.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:21.240 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:21.282 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:21.283 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:21.516 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507538.853263_65.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:18.853263Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:32:21.517 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:32:21.518 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:32:21.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:21.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:21.521 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:21.565 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:21.565 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:21.825 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507538.853263_65.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:18.853263Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:32:21.826 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:32:21.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:21.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:21.828 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:21.869 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:21.870 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:22.106 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.9.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.9.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.8.5"},{"key":8,"val":"00000000001.22.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:32:22.107 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-10 2020-07-23T12:32:22.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:22.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:22.109 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:22.157 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:22.158 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:22.353 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.22.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:32:22.354 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-10 2020-07-23T12:32:22.355 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-10 zones={test-zone2, test-zone1} 2020-07-23T12:32:23.311 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:32:23.324 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-23T12:32:23.325 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-23T12:32:23.325 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-23T12:32:23.326 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:23.326 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:32:23.336 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-23T12:32:23.337 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-23T12:32:23.338 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-23T12:32:23.339 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:23.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:23.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:23.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:23.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:23.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:23.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:23.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:23.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:24.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:24.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:24.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:24.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:24.491 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-10 zones={test-zone2, test-zone1} 2020-07-23T12:32:24.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:24.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:24.494 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:24.533 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:24.533 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:24.773 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.9.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.9.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.8.5"},{"key":8,"val":"00000000001.22.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:32:24.773 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-10 2020-07-23T12:32:24.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:24.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:24.775 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:24.857 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:24.858 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:25.111 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.22.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:32:25.112 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-10 2020-07-23T12:32:25.112 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-10 zones={test-zone3, test-zone1} 2020-07-23T12:32:25.380 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:32:25.387 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-23T12:32:25.388 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-23T12:32:25.388 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-23T12:32:25.389 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:25.390 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:32:25.397 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-23T12:32:25.398 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-23T12:32:25.399 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-23T12:32:25.399 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:25.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:25.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:25.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:25.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:25.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:25.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:25.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:26.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:26.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:26.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:26.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:26.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:26.744 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-10 zones={test-zone3, test-zone1} 2020-07-23T12:32:26.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:26.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:26.747 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:26.787 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:26.788 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:27.021 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.9.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.9.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.8.5"},{"key":8,"val":"00000000001.22.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:32:27.021 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-10 2020-07-23T12:32:27.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:27.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:27.024 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:27.070 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:27.070 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:27.339 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.9.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.9.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.8.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.22.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:32:27.340 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-10 2020-07-23T12:32:27.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:27.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:27.342 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:27.435 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:27.436 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:27.627 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.7.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.16.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.8.5"},{"key":8,"val":"00000000001.7.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:32:27.627 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-11 2020-07-23T12:32:27.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:27.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:27.629 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:27.681 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:27.681 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:27.947 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:32:27.948 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-11 2020-07-23T12:32:27.949 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-11 zones={test-zone1, test-zone2} 2020-07-23T12:32:29.084 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:32:29.098 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-23T12:32:29.099 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-23T12:32:29.099 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-23T12:32:29.100 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:29.100 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:32:29.107 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-23T12:32:29.108 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-23T12:32:29.109 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-23T12:32:29.109 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:29.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:29.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:29.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:29.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:29.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:29.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:29.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:29.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:29.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:29.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:30.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:30.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:30.229 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-11 zones={test-zone1, test-zone2} 2020-07-23T12:32:30.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:30.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:30.231 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:30.269 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:30.269 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:30.501 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.7.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.16.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.8.5"},{"key":8,"val":"00000000001.7.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:32:30.501 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-11 2020-07-23T12:32:30.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:30.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:30.503 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:30.552 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:30.552 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:30.810 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:32:30.811 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-11 2020-07-23T12:32:30.812 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-11 zones={test-zone3, test-zone2} 2020-07-23T12:32:30.845 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:32:30.852 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-23T12:32:30.852 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-23T12:32:30.853 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-23T12:32:30.853 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:30.854 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:32:30.861 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-23T12:32:30.861 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-23T12:32:30.862 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-23T12:32:30.862 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:30.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:30.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:31.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:31.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:31.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:31.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:31.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:31.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:31.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:31.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:31.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:31.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:32.105 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-11 zones={test-zone3, test-zone2} 2020-07-23T12:32:32.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:32.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:32.107 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:32.148 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:32.148 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:32.398 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.16.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.13.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.15.5"},{"key":8,"val":"00000000001.9.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:32:32.399 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-12 2020-07-23T12:32:32.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:32.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:32.402 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:32.486 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:32.486 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:32.738 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:32:32.739 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-12 2020-07-23T12:32:32.739 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-12 zones={test-zone1, test-zone3} 2020-07-23T12:32:33.581 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:32:33.590 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-23T12:32:33.591 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-23T12:32:33.591 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-23T12:32:33.592 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:33.596 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:32:33.603 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-23T12:32:33.604 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-23T12:32:33.605 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-23T12:32:33.606 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:33.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:33.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:33.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:33.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:33.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:34.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:34.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:34.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:34.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:34.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:34.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:34.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:34.855 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-12 zones={test-zone1, test-zone3} 2020-07-23T12:32:34.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:34.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:34.858 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:34.898 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:34.899 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:35.120 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.16.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.13.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.15.5"},{"key":8,"val":"00000000001.9.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:32:35.121 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-12 2020-07-23T12:32:35.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:35.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:35.123 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-12 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:35.169 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:35.169 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:35.405 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:32:35.406 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-12 2020-07-23T12:32:35.407 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-12 zones={test-zone2, test-zone3} 2020-07-23T12:32:35.762 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:32:35.770 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-23T12:32:35.771 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-23T12:32:35.772 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-23T12:32:35.773 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:35.774 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:32:35.784 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-07-23T12:32:35.785 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-07-23T12:32:35.786 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-07-23T12:32:35.787 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:35.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:35.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:36.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-07-23T12:32:36.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:36.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:36.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-07-23T12:32:36.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:36.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:36.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-07-23T12:32:36.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:36.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:36.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:37.015 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-12 zones={test-zone2, test-zone3} 2020-07-23T12:32:37.016 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2020-07-23T12:32:37.022 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-13 2020-07-23T12:32:37.788 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-14 2020-07-23T12:32:37.848 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-15 2020-07-23T12:32:38.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:38.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:38.077 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:38.121 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:38.135 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:38.333 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:32:38.364 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595507528.176421_165.1","last_update":"2020-07-23T12:32:08.176421Z"},{"marker":"1_1595507557.889012_83.1","last_update":"2020-07-23T12:32:37.889012Z"}] 2020-07-23T12:32:38.365 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:32:38.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:38.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:38.367 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:38.410 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:38.411 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:38.640 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507557.889012_83.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:37.889012Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:32:38.641 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:32:38.642 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:32:38.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:38.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:38.645 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:38.685 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:38.687 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:38.923 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507557.889012_83.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:37.889012Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:32:38.923 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:32:38.924 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:32:38.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:38.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:38.927 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:38.968 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:38.968 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:39.233 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507557.889012_83.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:37.889012Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:32:39.234 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:32:39.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:39.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:39.238 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:39.289 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:39.290 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:39.492 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.21.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:32:39.492 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-13 2020-07-23T12:32:39.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:39.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:39.509 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:39.558 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:39.559 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:39.805 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.21.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:32:39.806 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-13 2020-07-23T12:32:39.806 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-13 zones={test-zone2, test-zone1} 2020-07-23T12:32:40.707 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:32:40.715 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:40.715 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:32:40.767 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:40.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:40.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:41.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:41.112 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-13 zones={test-zone2, test-zone1} 2020-07-23T12:32:41.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:41.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:41.114 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:41.154 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:41.155 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:41.380 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.21.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:32:41.381 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-13 2020-07-23T12:32:41.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:41.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:41.386 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:41.465 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:41.465 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:41.721 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.21.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:32:41.722 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-13 2020-07-23T12:32:41.723 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-13 zones={test-zone3, test-zone1} 2020-07-23T12:32:41.836 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:32:41.882 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:41.883 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:32:41.894 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:41.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:42.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:42.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:42.261 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-13 zones={test-zone3, test-zone1} 2020-07-23T12:32:42.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:42.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:42.280 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:42.323 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:42.323 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:42.566 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.21.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:32:42.567 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-13 2020-07-23T12:32:42.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:42.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:42.569 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:42.613 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:42.614 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:42.888 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.21.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:32:42.889 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-13 2020-07-23T12:32:42.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:42.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:42.891 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:42.973 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:42.974 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:43.199 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.7.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:32:43.200 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-14 2020-07-23T12:32:43.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:43.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:43.204 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:43.264 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:43.264 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:43.488 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:32:43.488 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-14 2020-07-23T12:32:43.489 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-14 zones={test-zone1, test-zone2} 2020-07-23T12:32:44.547 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:32:44.559 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:44.560 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:32:44.610 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:44.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:44.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:44.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:44.962 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-14 zones={test-zone1, test-zone2} 2020-07-23T12:32:44.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:44.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:44.964 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:45.005 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:45.005 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:45.233 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.7.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:32:45.233 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-14 2020-07-23T12:32:45.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:45.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:45.235 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:45.279 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:45.279 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:45.523 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:32:45.523 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-14 2020-07-23T12:32:45.524 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-14 zones={test-zone3, test-zone2} 2020-07-23T12:32:45.624 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:32:45.674 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:45.675 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:32:45.684 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:45.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:45.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:45.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:46.053 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-14 zones={test-zone3, test-zone2} 2020-07-23T12:32:46.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:46.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:46.056 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:46.099 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:46.100 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:46.344 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.11.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:32:46.345 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-15 2020-07-23T12:32:46.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:46.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:46.348 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:46.429 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:46.430 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:46.688 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.11.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:32:46.689 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-15 2020-07-23T12:32:46.690 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-15 zones={test-zone1, test-zone3} 2020-07-23T12:32:47.844 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:32:47.852 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:47.853 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:32:47.898 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:47.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:48.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:48.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:48.243 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-15 zones={test-zone1, test-zone3} 2020-07-23T12:32:48.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:48.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:48.245 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:48.291 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:48.292 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:48.524 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.11.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:32:48.525 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-15 2020-07-23T12:32:48.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:48.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:48.527 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:48.569 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:48.570 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:48.816 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.11.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:32:48.816 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-15 2020-07-23T12:32:48.817 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-15 zones={test-zone2, test-zone3} 2020-07-23T12:32:50.634 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:32:50.653 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:50.667 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:32:50.718 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-07-23T12:32:50.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:50.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:50.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-07-23T12:32:51.077 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-15 zones={test-zone2, test-zone3} 2020-07-23T12:32:51.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:51.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:51.443 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:51.481 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:51.482 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:51.710 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.23.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:32:51.711 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-13 2020-07-23T12:32:51.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:51.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:51.713 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:51.755 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:51.756 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:51.991 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.21.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:32:51.992 WARNING:rgw_multi.tests:bucket pdivcf-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.23.5 target=00000000001.21.5 2020-07-23T12:32:56.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:56.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:56.995 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:57.039 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:57.039 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:57.266 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.23.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:32:57.267 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-13 2020-07-23T12:32:57.267 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-13 zones={test-zone2, test-zone1} 2020-07-23T12:32:58.847 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:32:58.855 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:32:58.967 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-13 zones={test-zone2, test-zone1} 2020-07-23T12:32:58.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:58.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:58.970 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:59.022 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:59.022 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:59.255 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.23.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:32:59.256 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-13 2020-07-23T12:32:59.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:59.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:32:59.258 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:32:59.301 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:32:59.301 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:32:59.561 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.23.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:32:59.562 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-13 2020-07-23T12:32:59.562 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-13 zones={test-zone3, test-zone1} 2020-07-23T12:33:00.188 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:33:00.195 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:33:00.302 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-13 zones={test-zone3, test-zone1} 2020-07-23T12:33:00.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:00.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:00.305 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:00.348 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:00.348 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:00.592 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.23.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:33:00.592 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-13 2020-07-23T12:33:00.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:00.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:00.594 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:00.639 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:00.640 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:00.912 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.23.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:00.912 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-13 2020-07-23T12:33:01.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:01.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:01.925 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:01.963 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:01.963 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:02.200 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.11.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:33:02.201 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-14 2020-07-23T12:33:02.202 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:02.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:02.204 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:02.249 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:02.250 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:02.503 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:02.504 WARNING:rgw_multi.tests:bucket pdivcf-14 zone test-zone1 behind zone test-zone2: shard=1 master=00000000002.11.5 target=00000000001.7.5 2020-07-23T12:33:07.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:07.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:07.511 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:07.555 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:07.555 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.11.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:07.816 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-14 2020-07-23T12:33:07.816 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-14 zones={test-zone1, test-zone2} 2020-07-23T12:33:09.618 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:33:09.628 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:33:09.738 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-14 zones={test-zone1, test-zone2} 2020-07-23T12:33:09.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:09.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:09.742 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:09.785 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:09.786 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:10.011 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.11.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:33:10.012 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-14 2020-07-23T12:33:10.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:10.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:10.013 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:10.054 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:10.054 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:10.300 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.11.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:10.300 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-14 2020-07-23T12:33:10.301 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-14 zones={test-zone3, test-zone2} 2020-07-23T12:33:11.181 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:33:11.230 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:33:11.297 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-14 zones={test-zone3, test-zone2} 2020-07-23T12:33:11.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:11.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:11.324 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:11.363 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:11.363 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:11.604 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.16.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:33:11.605 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-15 2020-07-23T12:33:11.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:11.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:11.607 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:11.688 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:11.689 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:11.958 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.11.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:11.959 WARNING:rgw_multi.tests:bucket pdivcf-15 zone test-zone1 behind zone test-zone3: shard=1 master=00000000002.16.5 target=00000000001.11.5 2020-07-23T12:33:16.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:16.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:16.963 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:17.008 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:17.008 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:17.274 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.16.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:17.275 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-15 2020-07-23T12:33:17.276 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-15 zones={test-zone1, test-zone3} 2020-07-23T12:33:18.081 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:33:18.089 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:33:18.198 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-15 zones={test-zone1, test-zone3} 2020-07-23T12:33:18.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:18.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:18.200 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:18.242 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:18.242 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:18.486 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.16.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:33:18.487 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-15 2020-07-23T12:33:18.487 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:18.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:18.489 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:18.538 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:18.538 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:18.784 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.16.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:18.785 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-15 2020-07-23T12:33:18.786 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-15 zones={test-zone2, test-zone3} 2020-07-23T12:33:20.170 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:33:20.175 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:33:20.282 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-15 zones={test-zone2, test-zone3} 2020-07-23T12:33:20.284 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2020-07-23T12:33:20.285 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-16 2020-07-23T12:33:20.490 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-17 2020-07-23T12:33:20.546 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-18 2020-07-23T12:33:20.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:20.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:20.656 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:20.695 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:20.695 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:20.892 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:33:20.926 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595507528.176421_165.1","last_update":"2020-07-23T12:32:08.176421Z"},{"marker":"1_1595507600.647133_114.1","last_update":"2020-07-23T12:33:20.647133Z"}] 2020-07-23T12:33:20.927 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:33:20.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:20.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:20.930 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:20.972 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:20.973 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:21.217 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507557.889012_83.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:37.889012Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:33:21.218 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595507600.647133_114.1 target=1_1595507557.889012_83.1 2020-07-23T12:33:26.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:26.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:26.221 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:26.263 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:26.263 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:26.487 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507600.647133_114.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:33:20.647133Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:33:26.488 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:33:26.488 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:33:26.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:26.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:26.490 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:26.530 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:26.531 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:26.775 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507600.647133_114.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:33:20.647133Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:33:26.836 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:33:26.837 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:33:26.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:26.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:26.839 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:26.905 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:26.906 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:27.178 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507600.647133_114.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:33:20.647133Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:33:27.179 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:33:27.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:27.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:27.464 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:27.505 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:27.506 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:27.731 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.31.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:33:27.732 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:33:27.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:27.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:27.734 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:27.779 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:27.779 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:28.039 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:28.040 WARNING:rgw_multi.tests:bucket pdivcf-16 zone test-zone2 behind zone test-zone1: shard=5 master=00000000002.31.10 target= 2020-07-23T12:33:33.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:33.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:33.044 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:33.083 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:33.083 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:33.337 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"FeHAXOfbLMeB9OY9u0M9LfhENOjzoMh","ns":""},"count":1},"inc_marker":{"position":"00000000005.37.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:33.338 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:33:33.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:33.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:33.339 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:33.387 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:33.399 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:33.632 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.37.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:33:33.632 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:33:33.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:33.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:33.634 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:33.715 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:33.715 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:33.979 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"FeHAXOfbLMeB9OY9u0M9LfhENOjzoMh","ns":""},"count":1},"inc_marker":{"position":"00000000005.37.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:33.979 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:33:33.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:33.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:33.981 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:34.060 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:34.061 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:34.308 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.37.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:33:34.309 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:33:34.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:34.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:34.313 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:34.359 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:34.359 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:34.632 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.37.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:34.633 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:33:35.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:35.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:35.340 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:35.382 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:35.383 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:35.603 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.12.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:33:35.603 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-17 2020-07-23T12:33:35.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:35.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:35.605 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:35.650 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:35.651 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:35.905 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:35.906 WARNING:rgw_multi.tests:bucket pdivcf-17 zone test-zone1 behind zone test-zone2: shard=5 master=00000000002.12.10 target= 2020-07-23T12:33:40.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:40.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:40.911 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:40.956 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:40.956 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:41.197 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"9HT9-CI78IDqiWMionVmK6prVf8fURD","ns":""},"count":1},"inc_marker":{"position":"00000000006.20.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:41.198 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-17 2020-07-23T12:33:41.199 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:41.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:41.201 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:41.243 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:41.243 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:41.469 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.20.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:33:41.470 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-17 2020-07-23T12:33:41.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:41.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:41.472 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:41.516 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:41.516 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:41.778 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"9HT9-CI78IDqiWMionVmK6prVf8fURD","ns":""},"count":1},"inc_marker":{"position":"00000000006.20.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:41.778 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-17 2020-07-23T12:33:41.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:41.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:41.872 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:41.912 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:41.913 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:42.160 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.9.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:33:42.161 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-18 2020-07-23T12:33:42.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:42.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:42.162 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:42.248 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:42.248 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:42.502 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:42.503 WARNING:rgw_multi.tests:bucket pdivcf-18 zone test-zone1 behind zone test-zone3: shard=5 master=00000000002.9.10 target= 2020-07-23T12:33:47.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:47.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:47.507 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:47.554 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:47.555 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:47.823 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"p.DXFqk53J5UMoLG4nxXT3rbPqbiGll","ns":""},"count":1},"inc_marker":{"position":"00000000005.15.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:47.823 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-18 2020-07-23T12:33:47.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:47.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:47.831 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:47.907 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:47.908 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:48.132 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.15.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:33:48.133 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-18 2020-07-23T12:33:48.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:48.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:48.134 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:48.180 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:48.182 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:48.454 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"p.DXFqk53J5UMoLG4nxXT3rbPqbiGll","ns":""},"count":1},"inc_marker":{"position":"00000000005.15.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:48.454 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-18 2020-07-23T12:33:53.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:53.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:53.372 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:53.406 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:53.406 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:53.634 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.37.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.15.9"},{"key":4,"val":""},{"key":5,"val":"00000000008.43.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:33:53.635 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:33:53.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:53.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:53.637 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:53.682 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:53.682 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:53.948 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"wGUSlj6xKf2XuVKSWNC0OttFHADCJuW","ns":""},"count":1},"inc_marker":{"position":"00000000010.37.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gWuXAeDc2PsnQ3fJI5Rtgvnkv0q8KJ3","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"FeHAXOfbLMeB9OY9u0M9LfhENOjzoMh","ns":""},"count":1},"inc_marker":{"position":"00000000005.37.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:53.949 WARNING:rgw_multi.tests:bucket pdivcf-16 zone test-zone2 behind zone test-zone1: shard=5 master=00000000008.43.9 target=00000000005.37.9 2020-07-23T12:33:58.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:58.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:58.953 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:58.998 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:58.999 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:59.256 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"wGUSlj6xKf2XuVKSWNC0OttFHADCJuW","ns":""},"count":1},"inc_marker":{"position":"00000000010.37.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gWuXAeDc2PsnQ3fJI5Rtgvnkv0q8KJ3","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"FeHAXOfbLMeB9OY9u0M9LfhENOjzoMh","ns":""},"count":1},"inc_marker":{"position":"00000000008.43.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:59.257 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:33:59.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:59.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:59.260 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:59.302 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:59.303 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:59.543 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.37.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.15.9"},{"key":4,"val":""},{"key":5,"val":"00000000008.43.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:33:59.544 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:33:59.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:59.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:59.546 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:59.625 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:59.626 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:33:59.874 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"wGUSlj6xKf2XuVKSWNC0OttFHADCJuW","ns":""},"count":1},"inc_marker":{"position":"00000000010.37.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gWuXAeDc2PsnQ3fJI5Rtgvnkv0q8KJ3","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"FeHAXOfbLMeB9OY9u0M9LfhENOjzoMh","ns":""},"count":1},"inc_marker":{"position":"00000000008.43.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:33:59.875 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:33:59.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:59.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:33:59.877 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:33:59.959 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:33:59.960 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:00.178 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.37.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.15.9"},{"key":4,"val":""},{"key":5,"val":"00000000008.43.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:00.178 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:34:00.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:00.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:00.180 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:00.225 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:00.225 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:00.492 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.37.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.15.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.43.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:00.493 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:34:00.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:00.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:00.496 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:00.537 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:00.537 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:00.776 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.28.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.28.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.32.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:00.777 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-16 2020-07-23T12:34:00.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:00.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:00.780 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:00.842 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:00.843 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:01.113 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"wGUSlj6xKf2XuVKSWNC0OttFHADCJuW","ns":""},"count":1},"inc_marker":{"position":"00000000003.28.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gWuXAeDc2PsnQ3fJI5Rtgvnkv0q8KJ3","ns":""},"count":1},"inc_marker":{"position":"00000000010.28.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"FeHAXOfbLMeB9OY9u0M9LfhENOjzoMh","ns":""},"count":1},"inc_marker":{"position":"00000000007.32.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:01.114 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-16 2020-07-23T12:34:01.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:01.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:01.117 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:01.167 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:01.167 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:01.398 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.28.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.28.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.32.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:01.398 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-16 2020-07-23T12:34:01.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:01.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:01.400 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:01.454 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:01.455 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:01.703 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"wGUSlj6xKf2XuVKSWNC0OttFHADCJuW","ns":""},"count":1},"inc_marker":{"position":"00000000003.28.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gWuXAeDc2PsnQ3fJI5Rtgvnkv0q8KJ3","ns":""},"count":2},"inc_marker":{"position":"00000000010.28.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"FeHAXOfbLMeB9OY9u0M9LfhENOjzoMh","ns":""},"count":1},"inc_marker":{"position":"00000000007.32.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:01.704 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-16 2020-07-23T12:34:01.704 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:01.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:01.705 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:01.797 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:01.798 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:02.046 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.26.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.15.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.47.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:02.047 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-16 2020-07-23T12:34:02.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:02.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:02.049 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:02.134 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:02.134 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:02.383 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"wGUSlj6xKf2XuVKSWNC0OttFHADCJuW","ns":""},"count":1},"inc_marker":{"position":"00000000004.26.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gWuXAeDc2PsnQ3fJI5Rtgvnkv0q8KJ3","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"FeHAXOfbLMeB9OY9u0M9LfhENOjzoMh","ns":""},"count":1},"inc_marker":{"position":"00000000014.47.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:02.384 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-16 2020-07-23T12:34:02.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:02.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:02.389 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:02.468 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:02.469 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:02.695 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.26.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.15.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.47.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:02.695 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-16 2020-07-23T12:34:02.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:02.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:02.704 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:02.751 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:02.753 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:02.994 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"wGUSlj6xKf2XuVKSWNC0OttFHADCJuW","ns":""},"count":1},"inc_marker":{"position":"00000000004.26.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gWuXAeDc2PsnQ3fJI5Rtgvnkv0q8KJ3","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"FeHAXOfbLMeB9OY9u0M9LfhENOjzoMh","ns":""},"count":1},"inc_marker":{"position":"00000000014.47.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:02.995 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-16 2020-07-23T12:34:02.995 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-16 zones={test-zone2, test-zone1} 2020-07-23T12:34:04.198 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:34:04.213 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:04.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:04.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:04.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:04.216 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:34:04.227 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:04.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:04.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:04.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:04.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:04.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:04.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:04.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:04.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:04.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:04.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:04.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:04.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:04.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:05.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:05.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:05.298 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-16 zones={test-zone2, test-zone1} 2020-07-23T12:34:05.299 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-16 zones={test-zone3, test-zone1} 2020-07-23T12:34:06.233 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:34:06.240 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:06.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:06.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:06.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:06.244 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:34:06.249 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:06.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:06.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:06.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:06.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:06.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:06.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:06.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:06.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:06.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:06.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:06.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:06.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:06.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:07.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:07.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:07.265 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-16 zones={test-zone3, test-zone1} 2020-07-23T12:34:07.266 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-16 zones={test-zone3, test-zone2} 2020-07-23T12:34:07.641 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:34:07.652 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:07.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:07.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:07.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:07.654 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:34:07.663 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:07.664 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:07.664 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:07.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:07.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:07.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:07.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:07.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:07.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:08.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:08.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:08.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:08.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:08.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:08.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:08.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:08.689 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-16 zones={test-zone3, test-zone2} 2020-07-23T12:34:08.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:08.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:08.692 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:08.735 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:08.735 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:08.989 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.25.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.10.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.30.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:08.990 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-17 2020-07-23T12:34:08.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:08.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:08.992 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:09.039 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:09.039 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:09.279 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a80Vd2US8r20ccux8OLxkXFAh.57JUC","ns":""},"count":1},"inc_marker":{"position":"00000000010.25.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"yi7fSHPIbfNfl.jnzTz8CVImFiQF3yr","ns":""},"count":1},"inc_marker":{"position":"00000000003.10.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"9HT9-CI78IDqiWMionVmK6prVf8fURD","ns":""},"count":1},"inc_marker":{"position":"00000000007.30.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:09.280 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-17 2020-07-23T12:34:09.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:09.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:09.282 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:09.322 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:09.322 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:09.557 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.25.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.10.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.30.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:09.558 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-17 2020-07-23T12:34:09.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:09.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:09.561 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:09.650 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:09.651 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:09.913 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a80Vd2US8r20ccux8OLxkXFAh.57JUC","ns":""},"count":1},"inc_marker":{"position":"00000000010.25.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"yi7fSHPIbfNfl.jnzTz8CVImFiQF3yr","ns":""},"count":1},"inc_marker":{"position":"00000000003.10.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"9HT9-CI78IDqiWMionVmK6prVf8fURD","ns":""},"count":1},"inc_marker":{"position":"00000000007.30.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:09.914 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-17 2020-07-23T12:34:09.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:09.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:09.916 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:09.997 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:09.998 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:10.230 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.25.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.10.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.30.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:10.230 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-17 2020-07-23T12:34:10.231 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:10.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:10.232 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:10.279 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:10.279 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:10.553 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.25.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.10.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.30.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:10.553 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-17 2020-07-23T12:34:10.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:10.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:10.555 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:10.590 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:10.591 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:10.807 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.13.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.32.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.26.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:10.807 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-17 2020-07-23T12:34:10.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:10.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:10.809 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:10.853 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:10.854 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:11.106 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a80Vd2US8r20ccux8OLxkXFAh.57JUC","ns":""},"count":1},"inc_marker":{"position":"00000000003.13.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"yi7fSHPIbfNfl.jnzTz8CVImFiQF3yr","ns":""},"count":1},"inc_marker":{"position":"00000000010.32.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"9HT9-CI78IDqiWMionVmK6prVf8fURD","ns":""},"count":1},"inc_marker":{"position":"00000000009.26.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:11.107 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-17 2020-07-23T12:34:11.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:11.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:11.108 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:11.150 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:11.150 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:11.392 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.13.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.32.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.26.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:11.393 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-17 2020-07-23T12:34:11.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:11.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:11.395 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:11.438 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:11.438 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:11.699 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a80Vd2US8r20ccux8OLxkXFAh.57JUC","ns":""},"count":1},"inc_marker":{"position":"00000000003.13.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"yi7fSHPIbfNfl.jnzTz8CVImFiQF3yr","ns":""},"count":1},"inc_marker":{"position":"00000000010.32.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"9HT9-CI78IDqiWMionVmK6prVf8fURD","ns":""},"count":1},"inc_marker":{"position":"00000000009.26.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:11.699 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-17 2020-07-23T12:34:11.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:11.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:11.701 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:11.781 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:11.782 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:12.028 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.14.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.14.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.38.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:12.028 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-17 2020-07-23T12:34:12.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:12.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:12.030 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:12.110 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:12.111 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:12.371 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a80Vd2US8r20ccux8OLxkXFAh.57JUC","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"yi7fSHPIbfNfl.jnzTz8CVImFiQF3yr","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"9HT9-CI78IDqiWMionVmK6prVf8fURD","ns":""},"count":1},"inc_marker":{"position":"00000000014.38.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:12.372 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-17 2020-07-23T12:34:12.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:12.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:12.374 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:12.452 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:12.453 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:12.678 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.14.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.14.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.38.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:12.679 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-17 2020-07-23T12:34:12.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:12.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:12.681 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:12.726 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:12.726 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:12.984 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a80Vd2US8r20ccux8OLxkXFAh.57JUC","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"yi7fSHPIbfNfl.jnzTz8CVImFiQF3yr","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"9HT9-CI78IDqiWMionVmK6prVf8fURD","ns":""},"count":1},"inc_marker":{"position":"00000000014.38.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:12.985 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-17 2020-07-23T12:34:12.986 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-17 zones={test-zone2, test-zone1} 2020-07-23T12:34:14.350 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:34:14.357 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:14.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:14.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:14.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:14.360 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:34:14.367 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:14.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:14.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:14.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:14.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:14.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:14.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:14.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:14.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:14.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:14.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:14.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:14.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:15.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:15.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:15.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:15.372 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-17 zones={test-zone2, test-zone1} 2020-07-23T12:34:15.373 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-17 zones={test-zone3, test-zone1} 2020-07-23T12:34:16.238 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:34:16.245 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:16.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:16.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:16.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:16.248 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:34:16.254 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:16.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:16.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:16.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:16.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:16.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:16.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:16.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:16.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:16.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:16.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:16.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:16.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:17.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:17.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:17.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:17.384 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-17 zones={test-zone3, test-zone1} 2020-07-23T12:34:17.385 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-17 zones={test-zone3, test-zone2} 2020-07-23T12:34:18.156 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:34:18.166 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:18.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:18.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:18.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:18.168 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:34:18.176 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:18.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:18.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:18.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:18.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:18.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:18.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:18.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:18.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:18.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:18.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:18.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:18.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:18.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:18.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:19.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:19.190 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-17 zones={test-zone3, test-zone2} 2020-07-23T12:34:19.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:19.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:19.193 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:19.234 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:19.235 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:19.477 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.37.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.33.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.19.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:19.478 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-18 2020-07-23T12:34:19.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:19.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:19.480 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:19.523 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:19.524 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:19.760 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sCE03qg7rn6r1LyZm-lgv8kF9qG0WDJ","ns":""},"count":1},"inc_marker":{"position":"00000000010.37.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"PKp8r-HYilmQgg-VpVoj7PdxCI1isYS","ns":""},"count":1},"inc_marker":{"position":"00000000003.33.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"p.DXFqk53J5UMoLG4nxXT3rbPqbiGll","ns":""},"count":1},"inc_marker":{"position":"00000000007.19.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:19.761 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-18 2020-07-23T12:34:19.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:19.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:19.764 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:19.807 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:19.808 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:20.039 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.37.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.33.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.19.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:20.040 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-18 2020-07-23T12:34:20.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:20.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:20.043 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:20.123 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:20.124 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:20.388 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sCE03qg7rn6r1LyZm-lgv8kF9qG0WDJ","ns":""},"count":1},"inc_marker":{"position":"00000000010.37.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"PKp8r-HYilmQgg-VpVoj7PdxCI1isYS","ns":""},"count":1},"inc_marker":{"position":"00000000003.33.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"p.DXFqk53J5UMoLG4nxXT3rbPqbiGll","ns":""},"count":1},"inc_marker":{"position":"00000000007.19.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:20.389 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-18 2020-07-23T12:34:20.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:20.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:20.392 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:20.480 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:20.493 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:20.706 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.37.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.33.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.19.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:20.706 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-18 2020-07-23T12:34:20.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:20.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:20.708 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:20.756 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:20.757 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:21.027 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.37.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.33.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.19.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:21.028 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-18 2020-07-23T12:34:21.029 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:21.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:21.030 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:21.111 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:21.111 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:21.341 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.18.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.42.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.17.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:21.342 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-18 2020-07-23T12:34:21.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:21.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:21.344 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:21.386 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:21.387 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:21.642 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sCE03qg7rn6r1LyZm-lgv8kF9qG0WDJ","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"PKp8r-HYilmQgg-VpVoj7PdxCI1isYS","ns":""},"count":1},"inc_marker":{"position":"00000000010.42.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"p.DXFqk53J5UMoLG4nxXT3rbPqbiGll","ns":""},"count":1},"inc_marker":{"position":"00000000007.17.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:21.643 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-18 2020-07-23T12:34:21.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:21.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:21.644 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:21.685 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:21.685 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:21.918 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.18.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.42.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.17.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:21.919 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-18 2020-07-23T12:34:21.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:21.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:21.921 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:21.966 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:21.967 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:22.220 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sCE03qg7rn6r1LyZm-lgv8kF9qG0WDJ","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"PKp8r-HYilmQgg-VpVoj7PdxCI1isYS","ns":""},"count":1},"inc_marker":{"position":"00000000010.42.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"p.DXFqk53J5UMoLG4nxXT3rbPqbiGll","ns":""},"count":1},"inc_marker":{"position":"00000000007.17.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:22.221 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-18 2020-07-23T12:34:22.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:22.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:22.223 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:22.263 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:22.263 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:22.510 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.16.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.30.9"},{"key":4,"val":""},{"key":5,"val":"00000000015.35.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:22.511 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-18 2020-07-23T12:34:22.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:22.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:22.512 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:22.598 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:22.598 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:22.865 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sCE03qg7rn6r1LyZm-lgv8kF9qG0WDJ","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"PKp8r-HYilmQgg-VpVoj7PdxCI1isYS","ns":""},"count":1},"inc_marker":{"position":"00000000003.30.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"p.DXFqk53J5UMoLG4nxXT3rbPqbiGll","ns":""},"count":1},"inc_marker":{"position":"00000000015.35.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:22.865 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-18 2020-07-23T12:34:22.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:22.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:22.867 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:22.950 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:22.951 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:23.185 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.16.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.30.9"},{"key":4,"val":""},{"key":5,"val":"00000000015.35.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:23.186 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-18 2020-07-23T12:34:23.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:23.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:23.188 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:23.234 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:23.235 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:23.492 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sCE03qg7rn6r1LyZm-lgv8kF9qG0WDJ","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"PKp8r-HYilmQgg-VpVoj7PdxCI1isYS","ns":""},"count":1},"inc_marker":{"position":"00000000003.30.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"p.DXFqk53J5UMoLG4nxXT3rbPqbiGll","ns":""},"count":1},"inc_marker":{"position":"00000000015.35.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:23.493 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-18 2020-07-23T12:34:23.493 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-18 zones={test-zone2, test-zone1} 2020-07-23T12:34:24.083 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:34:24.091 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:24.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:24.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:24.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:24.093 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:34:24.102 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:24.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:24.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:24.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:24.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:24.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:24.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:24.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:24.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:24.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:24.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:24.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:24.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:24.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:24.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:24.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:25.105 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-18 zones={test-zone2, test-zone1} 2020-07-23T12:34:25.106 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-18 zones={test-zone3, test-zone1} 2020-07-23T12:34:25.494 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:34:25.500 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:25.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:25.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:25.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:25.502 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:34:25.507 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:25.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:25.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:25.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:25.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:25.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:25.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:25.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:25.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:25.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:26.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:26.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:26.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:26.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:26.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:26.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:26.734 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-18 zones={test-zone3, test-zone1} 2020-07-23T12:34:26.735 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-18 zones={test-zone3, test-zone2} 2020-07-23T12:34:27.361 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:34:27.370 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:27.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:27.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:27.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:27.372 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:34:27.387 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:27.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:27.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:27.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:27.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:27.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:27.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:27.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:27.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:27.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:27.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:27.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:28.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:28.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:28.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:28.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:28.405 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-18 zones={test-zone3, test-zone2} 2020-07-23T12:34:29.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:29.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:29.740 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:29.779 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:29.780 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:30.006 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.43.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.15.9"},{"key":4,"val":""},{"key":5,"val":"00000000008.43.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:30.007 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:34:30.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:30.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:30.009 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:30.052 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:30.053 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:30.291 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"wGUSlj6xKf2XuVKSWNC0OttFHADCJuW","ns":""},"count":1},"inc_marker":{"position":"00000000010.37.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gWuXAeDc2PsnQ3fJI5Rtgvnkv0q8KJ3","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"FeHAXOfbLMeB9OY9u0M9LfhENOjzoMh","ns":""},"count":1},"inc_marker":{"position":"00000000008.43.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:30.291 WARNING:rgw_multi.tests:bucket pdivcf-16 zone test-zone2 behind zone test-zone1: shard=1 master=00000000011.43.5 target=00000000010.37.5 2020-07-23T12:34:35.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:35.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:35.294 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:35.337 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:35.338 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:35.567 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"wGUSlj6xKf2XuVKSWNC0OttFHADCJuW","ns":""},"count":1},"inc_marker":{"position":"00000000011.43.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gWuXAeDc2PsnQ3fJI5Rtgvnkv0q8KJ3","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"FeHAXOfbLMeB9OY9u0M9LfhENOjzoMh","ns":""},"count":1},"inc_marker":{"position":"00000000009.46.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:35.568 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:34:35.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:35.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:35.570 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:35.612 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:35.612 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:35.839 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.43.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.18.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.46.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:35.840 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:34:35.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:35.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:35.841 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:35.922 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:35.923 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:36.162 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"wGUSlj6xKf2XuVKSWNC0OttFHADCJuW","ns":""},"count":1},"inc_marker":{"position":"00000000011.43.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gWuXAeDc2PsnQ3fJI5Rtgvnkv0q8KJ3","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"FeHAXOfbLMeB9OY9u0M9LfhENOjzoMh","ns":""},"count":1},"inc_marker":{"position":"00000000009.46.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:36.163 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:34:36.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:36.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:36.165 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:36.247 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:36.248 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:36.480 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.43.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.18.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.46.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:36.481 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:34:36.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:36.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:36.483 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:36.525 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:36.526 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:36.782 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.43.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.18.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000009.46.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:36.783 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-16 2020-07-23T12:34:36.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:36.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:36.785 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:36.863 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:36.864 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:37.082 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.31.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.34.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.35.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:37.083 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-16 2020-07-23T12:34:37.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:37.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:37.084 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:37.127 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:37.127 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:37.372 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"wGUSlj6xKf2XuVKSWNC0OttFHADCJuW","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gWuXAeDc2PsnQ3fJI5Rtgvnkv0q8KJ3","ns":""},"count":1},"inc_marker":{"position":"00000000011.34.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"FeHAXOfbLMeB9OY9u0M9LfhENOjzoMh","ns":""},"count":1},"inc_marker":{"position":"00000000008.35.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:37.373 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-16 2020-07-23T12:34:37.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:37.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:37.376 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:37.418 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:37.419 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:37.644 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.31.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.34.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.35.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:37.645 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-16 2020-07-23T12:34:37.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:37.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:37.647 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:37.690 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:37.691 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:37.918 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"wGUSlj6xKf2XuVKSWNC0OttFHADCJuW","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gWuXAeDc2PsnQ3fJI5Rtgvnkv0q8KJ3","ns":""},"count":2},"inc_marker":{"position":"00000000011.34.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"FeHAXOfbLMeB9OY9u0M9LfhENOjzoMh","ns":""},"count":1},"inc_marker":{"position":"00000000008.35.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:37.918 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-16 2020-07-23T12:34:37.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:37.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:37.920 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:38.005 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:38.006 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:38.211 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000005.29.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.18.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.50.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:38.211 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-16 2020-07-23T12:34:38.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:38.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:38.213 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:38.299 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:38.300 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:38.555 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"wGUSlj6xKf2XuVKSWNC0OttFHADCJuW","ns":""},"count":1},"inc_marker":{"position":"00000000005.29.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gWuXAeDc2PsnQ3fJI5Rtgvnkv0q8KJ3","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"FeHAXOfbLMeB9OY9u0M9LfhENOjzoMh","ns":""},"count":1},"inc_marker":{"position":"00000000015.50.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:38.556 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-16 2020-07-23T12:34:38.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:38.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:38.558 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:38.596 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:38.597 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:38.834 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000005.29.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.18.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.50.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:38.835 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-16 2020-07-23T12:34:38.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:38.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:38.837 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:38.881 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:38.881 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:39.138 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"wGUSlj6xKf2XuVKSWNC0OttFHADCJuW","ns":""},"count":1},"inc_marker":{"position":"00000000005.29.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gWuXAeDc2PsnQ3fJI5Rtgvnkv0q8KJ3","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"FeHAXOfbLMeB9OY9u0M9LfhENOjzoMh","ns":""},"count":1},"inc_marker":{"position":"00000000015.50.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:39.139 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-16 2020-07-23T12:34:39.140 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-16 zones={test-zone2, test-zone1} 2020-07-23T12:34:40.399 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:34:40.408 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:40.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:40.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:40.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:40.410 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:34:40.422 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:40.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:40.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:40.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:40.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:40.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:40.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:40.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:40.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:40.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:40.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:40.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:41.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:41.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:41.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:41.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:41.492 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-16 zones={test-zone2, test-zone1} 2020-07-23T12:34:41.493 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-16 zones={test-zone3, test-zone1} 2020-07-23T12:34:42.467 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:34:42.473 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:42.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:42.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:42.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:42.475 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:34:42.480 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:42.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:42.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:42.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:42.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:42.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:42.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:42.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:42.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:42.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:42.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:42.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:43.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:43.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:43.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:43.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:43.467 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-16 zones={test-zone3, test-zone1} 2020-07-23T12:34:43.468 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-16 zones={test-zone3, test-zone2} 2020-07-23T12:34:44.379 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:34:44.390 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:44.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:44.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:44.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:44.393 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:34:44.400 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:44.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:44.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:44.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:44.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:44.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:44.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:44.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:44.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:44.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:44.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:44.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:45.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:45.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:45.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:45.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:45.451 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-16 zones={test-zone3, test-zone2} 2020-07-23T12:34:45.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:45.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:45.474 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:45.533 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:45.533 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:45.787 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.31.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.13.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.33.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:45.788 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-17 2020-07-23T12:34:45.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:45.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:45.789 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:45.834 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:45.834 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:46.097 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a80Vd2US8r20ccux8OLxkXFAh.57JUC","ns":""},"count":1},"inc_marker":{"position":"00000000011.31.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"yi7fSHPIbfNfl.jnzTz8CVImFiQF3yr","ns":""},"count":1},"inc_marker":{"position":"00000000004.13.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"9HT9-CI78IDqiWMionVmK6prVf8fURD","ns":""},"count":1},"inc_marker":{"position":"00000000008.33.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:46.097 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-17 2020-07-23T12:34:46.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:46.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:46.110 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:46.149 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:46.150 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:46.396 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.31.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.13.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.33.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:46.396 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-17 2020-07-23T12:34:46.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:46.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:46.398 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:46.488 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:46.488 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:46.767 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a80Vd2US8r20ccux8OLxkXFAh.57JUC","ns":""},"count":1},"inc_marker":{"position":"00000000011.31.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"yi7fSHPIbfNfl.jnzTz8CVImFiQF3yr","ns":""},"count":1},"inc_marker":{"position":"00000000004.13.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"9HT9-CI78IDqiWMionVmK6prVf8fURD","ns":""},"count":1},"inc_marker":{"position":"00000000008.33.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:46.768 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-17 2020-07-23T12:34:46.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:46.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:46.770 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:46.815 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:46.816 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:47.062 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.31.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.13.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.33.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:47.063 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-17 2020-07-23T12:34:47.064 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:47.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:47.065 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:47.112 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:47.113 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:47.380 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.31.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.13.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.33.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:47.380 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-17 2020-07-23T12:34:47.381 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:47.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:47.382 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:47.419 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:47.420 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:47.649 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.16.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.38.5"},{"key":4,"val":""},{"key":5,"val":"00000000010.29.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:47.649 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-17 2020-07-23T12:34:47.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:47.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:47.651 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:47.698 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:47.699 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:47.951 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a80Vd2US8r20ccux8OLxkXFAh.57JUC","ns":""},"count":1},"inc_marker":{"position":"00000000004.16.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"yi7fSHPIbfNfl.jnzTz8CVImFiQF3yr","ns":""},"count":1},"inc_marker":{"position":"00000000011.38.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"9HT9-CI78IDqiWMionVmK6prVf8fURD","ns":""},"count":1},"inc_marker":{"position":"00000000010.29.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:47.951 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-17 2020-07-23T12:34:47.952 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:47.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:47.954 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:47.994 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:47.995 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:48.248 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.16.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.38.5"},{"key":4,"val":""},{"key":5,"val":"00000000010.29.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:48.249 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-17 2020-07-23T12:34:48.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:48.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:48.251 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:48.305 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:48.306 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:48.570 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a80Vd2US8r20ccux8OLxkXFAh.57JUC","ns":""},"count":1},"inc_marker":{"position":"00000000004.16.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"yi7fSHPIbfNfl.jnzTz8CVImFiQF3yr","ns":""},"count":1},"inc_marker":{"position":"00000000011.38.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"9HT9-CI78IDqiWMionVmK6prVf8fURD","ns":""},"count":1},"inc_marker":{"position":"00000000010.29.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:48.570 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-17 2020-07-23T12:34:48.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:48.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:48.573 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:48.652 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:48.653 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:48.901 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.17.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.17.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.41.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:48.902 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-17 2020-07-23T12:34:48.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:48.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:48.904 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:48.992 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:48.993 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:49.247 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a80Vd2US8r20ccux8OLxkXFAh.57JUC","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"yi7fSHPIbfNfl.jnzTz8CVImFiQF3yr","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"9HT9-CI78IDqiWMionVmK6prVf8fURD","ns":""},"count":1},"inc_marker":{"position":"00000000015.41.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:49.248 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-17 2020-07-23T12:34:49.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:49.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:49.250 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:49.329 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:49.329 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:49.556 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.17.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.17.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.41.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:49.557 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-17 2020-07-23T12:34:49.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:49.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:49.559 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:49.601 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:49.601 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:49.873 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a80Vd2US8r20ccux8OLxkXFAh.57JUC","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"yi7fSHPIbfNfl.jnzTz8CVImFiQF3yr","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"9HT9-CI78IDqiWMionVmK6prVf8fURD","ns":""},"count":1},"inc_marker":{"position":"00000000015.41.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:49.874 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-17 2020-07-23T12:34:49.875 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-17 zones={test-zone2, test-zone1} 2020-07-23T12:34:51.192 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:34:51.201 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:51.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:51.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:51.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:51.203 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:34:51.212 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:51.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:51.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:51.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:51.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:51.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:51.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:51.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:51.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:51.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:51.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:51.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:51.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:51.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:52.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:52.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:52.255 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-17 zones={test-zone2, test-zone1} 2020-07-23T12:34:52.255 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-17 zones={test-zone3, test-zone1} 2020-07-23T12:34:52.681 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:34:52.687 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:52.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:52.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:52.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:52.689 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:34:52.694 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:52.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:52.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:52.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:52.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:52.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:52.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:52.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:52.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:53.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:53.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:53.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:53.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:53.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:53.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:53.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:53.684 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-17 zones={test-zone3, test-zone1} 2020-07-23T12:34:53.685 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-17 zones={test-zone3, test-zone2} 2020-07-23T12:34:54.126 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:34:54.135 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:54.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:54.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:54.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:54.137 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:34:54.144 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:34:54.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:34:54.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:34:54.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:34:54.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:54.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:54.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:34:54.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:54.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:54.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:34:54.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:54.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:54.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:34:54.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:54.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:55.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:34:55.148 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-17 zones={test-zone3, test-zone2} 2020-07-23T12:34:55.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:55.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:55.150 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:55.192 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:55.192 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:55.432 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.43.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.36.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.22.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:55.433 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-18 2020-07-23T12:34:55.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:55.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:55.435 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:55.478 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:55.478 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:55.727 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sCE03qg7rn6r1LyZm-lgv8kF9qG0WDJ","ns":""},"count":1},"inc_marker":{"position":"00000000011.43.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"PKp8r-HYilmQgg-VpVoj7PdxCI1isYS","ns":""},"count":1},"inc_marker":{"position":"00000000004.36.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"p.DXFqk53J5UMoLG4nxXT3rbPqbiGll","ns":""},"count":1},"inc_marker":{"position":"00000000008.22.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:55.728 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-18 2020-07-23T12:34:55.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:55.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:55.731 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:55.774 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:55.774 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:56.030 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.43.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.36.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.22.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:56.031 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-18 2020-07-23T12:34:56.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:56.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:56.033 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:56.122 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:56.123 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:56.385 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sCE03qg7rn6r1LyZm-lgv8kF9qG0WDJ","ns":""},"count":1},"inc_marker":{"position":"00000000011.43.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"PKp8r-HYilmQgg-VpVoj7PdxCI1isYS","ns":""},"count":1},"inc_marker":{"position":"00000000004.36.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"p.DXFqk53J5UMoLG4nxXT3rbPqbiGll","ns":""},"count":1},"inc_marker":{"position":"00000000008.22.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:56.386 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-18 2020-07-23T12:34:56.386 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:56.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:56.387 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:56.466 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:56.467 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:56.709 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.43.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.36.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.22.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:56.710 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-18 2020-07-23T12:34:56.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:56.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:56.711 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:56.756 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:56.756 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:57.025 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.43.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.36.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.22.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:57.025 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-18 2020-07-23T12:34:57.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:57.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:57.027 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:57.110 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:57.110 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:57.348 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.21.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.48.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.20.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:57.349 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-18 2020-07-23T12:34:57.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:57.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:57.352 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:57.396 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:57.396 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:57.673 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sCE03qg7rn6r1LyZm-lgv8kF9qG0WDJ","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"PKp8r-HYilmQgg-VpVoj7PdxCI1isYS","ns":""},"count":1},"inc_marker":{"position":"00000000011.48.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"p.DXFqk53J5UMoLG4nxXT3rbPqbiGll","ns":""},"count":1},"inc_marker":{"position":"00000000008.20.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:57.674 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-18 2020-07-23T12:34:57.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:57.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:57.675 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:57.715 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:57.715 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:57.958 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.21.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.48.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.20.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:57.959 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-18 2020-07-23T12:34:57.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:57.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:57.961 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:58.004 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:58.004 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:58.260 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sCE03qg7rn6r1LyZm-lgv8kF9qG0WDJ","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"PKp8r-HYilmQgg-VpVoj7PdxCI1isYS","ns":""},"count":1},"inc_marker":{"position":"00000000011.48.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"p.DXFqk53J5UMoLG4nxXT3rbPqbiGll","ns":""},"count":1},"inc_marker":{"position":"00000000008.20.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:58.261 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-18 2020-07-23T12:34:58.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:58.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:58.263 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:58.341 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:58.342 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:58.599 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.19.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.33.5"},{"key":4,"val":""},{"key":5,"val":"00000000016.38.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:58.600 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-18 2020-07-23T12:34:58.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:58.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:58.601 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:58.687 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:58.688 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:58.950 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sCE03qg7rn6r1LyZm-lgv8kF9qG0WDJ","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"PKp8r-HYilmQgg-VpVoj7PdxCI1isYS","ns":""},"count":1},"inc_marker":{"position":"00000000004.33.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"p.DXFqk53J5UMoLG4nxXT3rbPqbiGll","ns":""},"count":1},"inc_marker":{"position":"00000000016.38.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:58.951 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-18 2020-07-23T12:34:58.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:58.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:58.953 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:58.995 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:58.996 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:59.235 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.19.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.33.5"},{"key":4,"val":""},{"key":5,"val":"00000000016.38.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:34:59.236 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-18 2020-07-23T12:34:59.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:59.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:34:59.237 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:34:59.281 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:34:59.281 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:34:59.537 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"sCE03qg7rn6r1LyZm-lgv8kF9qG0WDJ","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"PKp8r-HYilmQgg-VpVoj7PdxCI1isYS","ns":""},"count":1},"inc_marker":{"position":"00000000004.33.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"p.DXFqk53J5UMoLG4nxXT3rbPqbiGll","ns":""},"count":1},"inc_marker":{"position":"00000000016.38.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:34:59.538 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-18 2020-07-23T12:34:59.539 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-18 zones={test-zone2, test-zone1} 2020-07-23T12:35:00.148 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:35:00.156 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:00.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:35:00.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:35:00.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:35:00.174 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:35:00.196 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:00.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:35:00.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:35:00.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:35:00.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:00.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:00.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:00.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:35:00.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:35:00.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:35:00.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:35:00.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:35:00.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:35:00.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:35:01.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:35:01.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:35:01.263 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-18 zones={test-zone2, test-zone1} 2020-07-23T12:35:01.264 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-18 zones={test-zone3, test-zone1} 2020-07-23T12:35:01.495 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:35:01.501 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:01.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:35:01.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:35:01.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:35:01.503 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:35:01.509 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:01.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:35:01.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:35:01.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:35:01.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:01.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:01.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:01.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:35:01.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:35:01.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:35:01.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:35:02.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:35:02.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:35:02.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:35:02.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:35:02.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:35:02.531 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-18 zones={test-zone3, test-zone1} 2020-07-23T12:35:02.532 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-18 zones={test-zone3, test-zone2} 2020-07-23T12:35:02.772 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:35:02.781 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:02.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:35:02.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:35:02.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:35:02.784 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:35:02.790 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:02.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-07-23T12:35:02.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-07-23T12:35:02.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-07-23T12:35:02.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:02.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:02.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:02.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:35:03.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:35:03.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-07-23T12:35:03.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:35:03.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:35:03.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-07-23T12:35:03.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:35:03.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:35:03.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-07-23T12:35:03.865 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-18 zones={test-zone3, test-zone2} 2020-07-23T12:35:03.867 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2020-07-23T12:35:04.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:04.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:04.661 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:04.699 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:04.699 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:04.904 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:35:04.934 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595507528.176421_165.1","last_update":"2020-07-23T12:32:08.176421Z"},{"marker":"1_1595507704.656778_189.1","last_update":"2020-07-23T12:35:04.656778Z"}] 2020-07-23T12:35:04.935 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:35:04.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:04.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:04.938 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:04.979 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:04.979 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:05.218 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507600.647133_114.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:33:20.647133Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:35:05.219 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595507704.656778_189.1 target=1_1595507600.647133_114.1 2020-07-23T12:35:10.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:10.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:10.222 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:10.262 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:10.262 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:10.500 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507704.656778_189.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:04.656778Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:35:10.501 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:35:10.502 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:35:10.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:10.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:10.503 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:10.544 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:10.544 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:10.781 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507704.656778_189.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:04.656778Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:35:10.782 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:35:10.782 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:35:10.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:10.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:10.784 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:10.826 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:10.827 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:11.101 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507704.656778_189.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:04.656778Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:35:11.102 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:35:11.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:11.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:11.896 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:11.934 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:11.935 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:12.168 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.7.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:35:12.169 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-19 2020-07-23T12:35:12.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:12.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:12.171 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:12.216 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:12.216 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:12.468 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:35:12.468 WARNING:rgw_multi.tests:bucket pdivcf-19 zone test-zone2 behind zone test-zone1: shard=5 master=00000000002.7.10 target= 2020-07-23T12:35:17.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:17.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:17.472 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:17.518 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:17.519 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:17.764 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"K-NiHKAH8oglRVO8qiZFGdkM0QBMhur","ns":""},"count":1},"inc_marker":{"position":"00000000005.13.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:35:17.765 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-19 2020-07-23T12:35:17.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:17.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:17.767 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:17.812 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:17.812 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:18.048 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.13.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:35:18.048 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-19 2020-07-23T12:35:18.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:18.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:18.051 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:18.138 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:18.139 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:18.399 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"K-NiHKAH8oglRVO8qiZFGdkM0QBMhur","ns":""},"count":1},"inc_marker":{"position":"00000000005.13.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:35:18.399 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-19 2020-07-23T12:35:18.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:18.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:18.401 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:18.482 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:18.483 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:18.721 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.13.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:35:18.721 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-19 2020-07-23T12:35:18.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:18.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:18.723 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:18.771 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:18.772 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:19.036 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.13.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:35:19.037 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-19 2020-07-23T12:35:19.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:19.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:19.040 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:19.082 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:19.083 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:19.309 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.14.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:35:19.310 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-19 2020-07-23T12:35:19.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:19.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:19.311 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:19.354 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:19.354 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:19.611 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"K-NiHKAH8oglRVO8qiZFGdkM0QBMhur","ns":""},"count":1},"inc_marker":{"position":"00000000004.14.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:35:19.612 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-19 2020-07-23T12:35:19.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:19.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:19.614 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:19.654 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:19.655 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:19.892 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.14.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:35:19.893 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-19 2020-07-23T12:35:19.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:19.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:19.896 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:19.975 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:19.976 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:20.212 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"K-NiHKAH8oglRVO8qiZFGdkM0QBMhur","ns":""},"count":1},"inc_marker":{"position":"00000000004.14.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:35:20.213 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-19 2020-07-23T12:35:20.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:20.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:20.218 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:20.304 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:20.305 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:20.552 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.15.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:35:20.553 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-19 2020-07-23T12:35:20.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:20.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:20.555 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:20.640 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:20.640 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:20.904 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"K-NiHKAH8oglRVO8qiZFGdkM0QBMhur","ns":""},"count":1},"inc_marker":{"position":"00000000004.15.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:35:20.905 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-19 2020-07-23T12:35:20.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:20.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:20.907 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:20.944 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:20.945 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:21.192 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.15.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:35:21.193 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-19 2020-07-23T12:35:21.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:21.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:21.195 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:21.242 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:21.243 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:21.500 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"K-NiHKAH8oglRVO8qiZFGdkM0QBMhur","ns":""},"count":1},"inc_marker":{"position":"00000000004.15.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:35:21.501 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-19 2020-07-23T12:35:21.502 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-19 zones={test-zone2, test-zone1} 2020-07-23T12:35:21.992 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:35:22.000 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:22.001 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:35:22.050 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:22.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:22.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:22.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:22.250 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-19 zones={test-zone2, test-zone1} 2020-07-23T12:35:22.251 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-19 zones={test-zone3, test-zone1} 2020-07-23T12:35:22.647 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:35:22.698 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:22.699 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:35:22.708 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:22.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:22.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:22.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:22.922 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-19 zones={test-zone3, test-zone1} 2020-07-23T12:35:22.923 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-19 zones={test-zone3, test-zone2} 2020-07-23T12:35:22.933 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:35:22.986 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:22.987 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:35:22.994 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:23.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:23.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:23.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:23.218 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-19 zones={test-zone3, test-zone2} 2020-07-23T12:35:25.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:25.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:25.067 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:25.124 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:25.125 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:25.330 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.13.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000040.140.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:35:25.330 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-19 2020-07-23T12:35:25.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:25.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:25.332 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:25.383 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:25.383 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:25.595 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"K-NiHKAH8oglRVO8qiZFGdkM0QBMhur","ns":""},"count":1},"inc_marker":{"position":"00000000005.13.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zEDBQQQAXE8nHUvTxhNf12ZJk-MEfa0","ns":""},"count":2},"inc_marker":{"position":"00000000040.140.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:35:25.596 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-19 2020-07-23T12:35:25.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:25.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:25.598 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:25.643 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:25.644 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:25.887 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.13.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000045.152.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:35:25.888 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-19 2020-07-23T12:35:25.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:25.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:25.890 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:25.933 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:25.933 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:26.190 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"K-NiHKAH8oglRVO8qiZFGdkM0QBMhur","ns":""},"count":1},"inc_marker":{"position":"00000000005.13.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zEDBQQQAXE8nHUvTxhNf12ZJk-MEfa0","ns":""},"count":2},"inc_marker":{"position":"00000000042.146.8"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:35:26.191 WARNING:rgw_multi.tests:bucket pdivcf-19 zone test-zone3 behind zone test-zone1: shard=8 master=00000000045.152.9 target=00000000042.146.8 2020-07-23T12:35:31.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:31.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:31.195 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:31.239 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:31.240 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:31.509 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"K-NiHKAH8oglRVO8qiZFGdkM0QBMhur","ns":""},"count":1},"inc_marker":{"position":"00000000005.13.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zEDBQQQAXE8nHUvTxhNf12ZJk-MEfa0","ns":""},"count":2},"inc_marker":{"position":"00000000046.154.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:35:31.510 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-19 2020-07-23T12:35:31.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:31.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:31.513 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:31.553 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:31.554 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:31.801 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.13.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000046.154.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:35:31.802 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-19 2020-07-23T12:35:31.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:31.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:31.805 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:31.850 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:31.850 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:32.124 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.13.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000046.154.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:35:32.125 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-19 2020-07-23T12:35:32.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:32.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:32.128 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:32.206 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:32.206 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:32.431 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.14.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000047.141.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:35:32.432 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-19 2020-07-23T12:35:32.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:32.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:32.434 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:32.480 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:32.481 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:32.750 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"K-NiHKAH8oglRVO8qiZFGdkM0QBMhur","ns":""},"count":1},"inc_marker":{"position":"00000000004.14.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zEDBQQQAXE8nHUvTxhNf12ZJk-MEfa0","ns":""},"count":5},"inc_marker":{"position":"00000000047.141.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:35:32.751 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-19 2020-07-23T12:35:32.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:32.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:32.753 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:32.800 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:32.801 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:33.027 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.14.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000047.141.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:35:33.028 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-19 2020-07-23T12:35:33.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:33.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:33.030 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:33.082 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:33.082 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:33.339 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"K-NiHKAH8oglRVO8qiZFGdkM0QBMhur","ns":""},"count":1},"inc_marker":{"position":"00000000004.14.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zEDBQQQAXE8nHUvTxhNf12ZJk-MEfa0","ns":""},"count":5},"inc_marker":{"position":"00000000047.141.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:35:33.340 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-19 2020-07-23T12:35:33.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:33.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:33.343 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:33.422 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:33.423 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:33.676 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.15.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000050.156.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:35:33.676 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-19 2020-07-23T12:35:33.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:33.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:33.678 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:33.763 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:33.763 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:34.018 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"K-NiHKAH8oglRVO8qiZFGdkM0QBMhur","ns":""},"count":1},"inc_marker":{"position":"00000000004.15.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"SpEgYJ2yGGm0LHu6b0E5JDq7h0vBqV6","ns":""},"count":2},"inc_marker":{"position":"00000000050.156.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:35:34.018 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-19 2020-07-23T12:35:34.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:34.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:34.020 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:34.057 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:34.058 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:34.295 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.15.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000050.156.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:35:34.296 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-19 2020-07-23T12:35:34.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:34.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:35:34.298 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:35:34.344 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:35:34.345 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:35:34.596 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"K-NiHKAH8oglRVO8qiZFGdkM0QBMhur","ns":""},"count":1},"inc_marker":{"position":"00000000004.15.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"SpEgYJ2yGGm0LHu6b0E5JDq7h0vBqV6","ns":""},"count":2},"inc_marker":{"position":"00000000050.156.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:35:34.597 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-19 2020-07-23T12:35:34.598 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-19 zones={test-zone2, test-zone1} 2020-07-23T12:35:36.324 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:35:36.336 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:36.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.350 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:35:36.360 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:36.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:36.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:36.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:36.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:36.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:36.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:36.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:36.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:36.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:37.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:37.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:37.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:37.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:37.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:37.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:37.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:37.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:37.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:37.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:38.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:38.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:38.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:38.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:38.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:38.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:38.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:38.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:38.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:38.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:39.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:39.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:39.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:39.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:39.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:39.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:39.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:39.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:39.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:39.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:40.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:40.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:40.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:40.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:40.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:40.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:40.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:40.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:40.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:40.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:41.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:41.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:41.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:41.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:41.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:41.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:41.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:41.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:41.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:41.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:41.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:42.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:42.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:42.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:42.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:42.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:42.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:42.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:42.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:42.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:43.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:43.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:43.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:43.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:43.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:43.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:43.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:43.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:43.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:43.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:44.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:44.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:44.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:44.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:44.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:44.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:44.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:44.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:44.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:45.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:45.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:45.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:45.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:45.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:45.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:45.619 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-19 zones={test-zone2, test-zone1} 2020-07-23T12:35:45.620 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-19 zones={test-zone3, test-zone1} 2020-07-23T12:35:46.283 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:35:46.293 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:46.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.318 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:35:46.329 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:46.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:46.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:46.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:46.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:46.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:46.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:46.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:46.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:46.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:46.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:47.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:47.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:47.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:47.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:47.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:47.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:47.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:47.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:47.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:48.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:48.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:48.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:48.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:48.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:48.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:48.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:48.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:48.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:48.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:49.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:49.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:49.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:49.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:49.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:49.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:49.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:49.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:49.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:49.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:50.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:50.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:50.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:50.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:50.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:50.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:50.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:50.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:50.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:51.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:51.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:51.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:51.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:51.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:51.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:51.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:51.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:51.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:51.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:52.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:52.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:52.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:52.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:52.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:52.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:52.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:52.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:52.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:52.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:53.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:53.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:53.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:53.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:53.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:53.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:53.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:53.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:53.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:53.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:54.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:54.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:54.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:54.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:54.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:54.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:54.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:54.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:54.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:54.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:55.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:55.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:55.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:55.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:55.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:55.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:55.704 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-19 zones={test-zone3, test-zone1} 2020-07-23T12:35:55.704 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-19 zones={test-zone3, test-zone2} 2020-07-23T12:35:56.370 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:35:56.382 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:56.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.400 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:35:56.411 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-07-23T12:35:56.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.421 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:35:56.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:56.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:56.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-07-23T12:35:56.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:56.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:56.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:56.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:56.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:57.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:57.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:57.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:57.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:57.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:57.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:57.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:57.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:57.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:57.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:58.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:58.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:58.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:58.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:58.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:58.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:58.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:58.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:58.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:58.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:59.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:59.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:59.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:59.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:59.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:59.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:59.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:59.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:59.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:35:59.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:00.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:00.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:00.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:00.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:00.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:00.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:00.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:00.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:00.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:00.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:01.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:01.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:01.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:01.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:01.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:01.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:01.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:01.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:01.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:01.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:02.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:02.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:02.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:02.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:02.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:02.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:02.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:02.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:02.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:02.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:03.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:03.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:03.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:03.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:03.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:03.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:03.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:03.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:03.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:04.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:04.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:04.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:04.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:04.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:04.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:04.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:04.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:04.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:04.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:05.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:05.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:05.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:05.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:05.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:05.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:05.664 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-19 zones={test-zone3, test-zone2} 2020-07-23T12:36:05.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:05.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:05.677 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:05.718 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:05.718 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:05.987 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1595507675.998941_262.2","last_update":"2020-07-23T12:34:35.998941Z"},{"marker":"1_1595507742.016237_164.1","last_update":"2020-07-23T12:35:42.016237Z"},{"marker":"1_1595507720.013682_138.1","last_update":"2020-07-23T12:35:20.013682Z"}] 2020-07-23T12:36:05.988 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-23T12:36:05.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:05.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:05.991 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:06.033 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:06.033 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:06.266 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":988729214735988972},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507675.998941_262.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:35.998941Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595507742.016237_164.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:42.016237Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595507720.013682_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:20.013682Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:36:06.267 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-23T12:36:06.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:06.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:06.268 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:06.308 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:06.309 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:06.563 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1595507675.998941_262.2","last_update":"2020-07-23T12:34:35.998941Z"},{"marker":"1_1595507742.016237_164.1","last_update":"2020-07-23T12:35:42.016237Z"},{"marker":"1_1595507720.013682_138.1","last_update":"2020-07-23T12:35:20.013682Z"}] 2020-07-23T12:36:06.563 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-23T12:36:06.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:06.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:06.565 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:06.650 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:06.650 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:06.880 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":5372564157323408779},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507675.998941_262.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:35.998941Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595507742.016237_164.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:42.016237Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595507720.013682_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:20.013682Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:36:06.881 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-23T12:36:06.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:06.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:06.883 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:06.962 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:06.962 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:07.211 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1595507675.998941_262.2","last_update":"2020-07-23T12:34:35.998941Z"},{"marker":"1_1595507742.016237_164.1","last_update":"2020-07-23T12:35:42.016237Z"},{"marker":"1_1595507720.013682_138.1","last_update":"2020-07-23T12:35:20.013682Z"}] 2020-07-23T12:36:07.211 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-23T12:36:07.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:07.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:07.214 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:07.256 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:07.257 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:07.524 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1496412237659215512},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507675.998941_262.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:35.998941Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595507742.016237_164.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:42.016237Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595507720.013682_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:20.013682Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:36:07.525 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-23T12:36:07.526 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:07.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:07.529 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:07.615 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:07.615 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:07.833 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1595507676.941349_266.3","last_update":"2020-07-23T12:34:36.941349Z"},{"marker":"1_1595507742.951709_129.1","last_update":"2020-07-23T12:35:42.951709Z"},{"marker":"1_1595507720.947303_140.1","last_update":"2020-07-23T12:35:20.947303Z"}] 2020-07-23T12:36:07.833 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-23T12:36:07.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:07.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:07.835 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:07.878 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:07.878 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:08.141 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14254013310941169947},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507676.941349_266.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:36.941349Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595507742.951709_129.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:42.951709Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595507720.947303_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:20.947303Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:36:08.141 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-23T12:36:08.142 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:08.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:08.143 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:08.190 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:08.190 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:08.469 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1595507676.941349_266.3","last_update":"2020-07-23T12:34:36.941349Z"},{"marker":"1_1595507742.951709_129.1","last_update":"2020-07-23T12:35:42.951709Z"},{"marker":"1_1595507720.947303_140.1","last_update":"2020-07-23T12:35:20.947303Z"}] 2020-07-23T12:36:08.481 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-23T12:36:08.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:08.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:08.483 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:08.528 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:08.529 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:08.780 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11610391562858473267},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507676.941349_266.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:36.941349Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595507742.951709_129.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:42.951709Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595507720.947303_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:20.947303Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:36:08.780 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-23T12:36:08.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:08.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:08.783 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:08.862 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:08.863 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:09.117 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1595507676.033316_221.3","last_update":"2020-07-23T12:34:36.033316Z"},{"marker":"1_1595507742.047544_120.1","last_update":"2020-07-23T12:35:42.047544Z"},{"marker":"1_1595507720.041484_113.1","last_update":"2020-07-23T12:35:20.041484Z"}] 2020-07-23T12:36:09.118 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-23T12:36:09.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:09.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:09.121 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:09.206 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:09.207 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:09.451 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14010726936006743351},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507676.033316_221.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:36.033316Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595507742.047544_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:42.047544Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595507720.041484_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:20.041484Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:36:09.459 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-23T12:36:09.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:09.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:09.461 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:09.532 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:09.533 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:09.775 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1595507676.033316_221.3","last_update":"2020-07-23T12:34:36.033316Z"},{"marker":"1_1595507742.047544_120.1","last_update":"2020-07-23T12:35:42.047544Z"},{"marker":"1_1595507720.041484_113.1","last_update":"2020-07-23T12:35:20.041484Z"}] 2020-07-23T12:36:09.776 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-23T12:36:09.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:09.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:09.779 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:09.822 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:09.822 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:10.070 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3438718294491381619},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507676.033316_221.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:36.033316Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595507742.047544_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:42.047544Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595507720.041484_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:20.041484Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:36:10.071 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-23T12:36:10.072 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2020-07-23T12:36:10.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:10.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:10.588 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:10.629 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:10.630 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:10.850 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:36:10.895 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595507528.176421_165.1","last_update":"2020-07-23T12:32:08.176421Z"},{"marker":"1_1595507770.579148_209.1","last_update":"2020-07-23T12:36:10.579148Z"}] 2020-07-23T12:36:10.895 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:36:10.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:10.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:10.899 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:10.949 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:10.950 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:11.190 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507704.656778_189.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:04.656778Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:36:11.191 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595507770.579148_209.1 target=1_1595507704.656778_189.1 2020-07-23T12:36:16.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:16.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:16.195 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:16.236 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:16.236 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:16.489 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507770.579148_209.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:36:10.579148Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:36:16.490 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:36:16.492 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:36:16.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:16.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:16.494 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:16.536 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:16.537 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:16.779 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507770.579148_209.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:36:10.579148Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:36:16.780 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:36:16.780 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:36:16.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:16.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:16.783 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:16.836 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:16.836 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:17.101 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507770.579148_209.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:36:10.579148Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:36:17.102 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:36:17.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:17.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:17.793 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:17.832 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:17.832 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:18.070 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.16.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:36:18.071 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:36:18.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:18.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:18.073 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:18.117 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:18.118 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:18.392 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:36:18.393 WARNING:rgw_multi.tests:bucket pdivcf-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000001.16.5 target= 2020-07-23T12:36:23.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:23.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:23.409 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:23.454 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:23.454 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:23.724 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:36:23.725 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:36:23.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:23.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:23.728 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:23.771 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:23.772 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:24.005 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.16.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:36:24.006 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:36:24.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:24.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:24.008 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:24.090 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:24.091 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:24.353 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:36:24.354 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:36:24.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:24.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:24.356 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:24.436 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:24.436 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:24.695 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.16.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:36:24.695 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:36:24.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:24.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:24.698 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:24.751 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:24.752 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:25.017 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.16.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:36:25.017 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:36:25.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:25.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:25.019 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:25.107 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:25.109 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:25.343 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.13.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:36:25.343 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-20 2020-07-23T12:36:25.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:25.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:25.345 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:25.391 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:25.392 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:25.665 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:36:25.665 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-20 2020-07-23T12:36:25.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:25.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:25.668 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:25.710 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:25.710 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:25.953 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.13.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:36:25.954 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-20 2020-07-23T12:36:25.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:25.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:25.956 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:26.004 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:26.005 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:26.258 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:36:26.259 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-20 2020-07-23T12:36:26.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:26.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:26.261 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:26.304 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:26.304 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:26.549 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.15.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:36:26.549 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-20 2020-07-23T12:36:26.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:26.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:26.551 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:26.635 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:26.635 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:26.907 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:36:26.908 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-20 2020-07-23T12:36:26.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:26.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:26.910 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:26.994 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:26.995 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:27.227 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.15.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:36:27.228 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-20 2020-07-23T12:36:27.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:27.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:27.232 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:27.278 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:27.278 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:27.515 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:36:27.516 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-20 2020-07-23T12:36:27.516 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-20 zones={test-zone2, test-zone1} 2020-07-23T12:36:28.875 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:36:28.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:28.885 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:36:28.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:28.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:29.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:29.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:29.168 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-20 zones={test-zone2, test-zone1} 2020-07-23T12:36:29.169 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-20 zones={test-zone3, test-zone1} 2020-07-23T12:36:29.730 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:36:29.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:29.783 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:36:29.792 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:29.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:29.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:29.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:30.022 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-20 zones={test-zone3, test-zone1} 2020-07-23T12:36:30.023 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-20 zones={test-zone3, test-zone2} 2020-07-23T12:36:30.037 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:36:30.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:30.105 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:36:30.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:30.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:30.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:30.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:30.366 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-20 zones={test-zone3, test-zone2} 2020-07-23T12:36:30.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:30.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:30.386 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:30.426 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:30.427 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:30.636 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:36:30.676 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595507528.176421_165.1","last_update":"2020-07-23T12:32:08.176421Z"},{"marker":"1_1595507790.379387_217.1","last_update":"2020-07-23T12:36:30.379387Z"}] 2020-07-23T12:36:30.677 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:36:30.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:30.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:30.681 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:30.738 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:30.738 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:30.992 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507770.579148_209.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:36:10.579148Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:36:30.993 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595507790.379387_217.1 target=1_1595507770.579148_209.1 2020-07-23T12:36:35.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:35.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:35.997 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:36.047 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:36.047 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:36.286 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507790.379387_217.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:36:30.379387Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:36:36.287 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:36:36.287 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:36:36.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:36.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:36.289 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:36.328 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:36.329 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:36.573 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507790.379387_217.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:36:30.379387Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:36:36.574 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:36:36.575 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:36:36.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:36.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:36.576 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:36.618 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:36.619 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:36.977 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507790.379387_217.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:36:30.379387Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:36:36.977 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:36:37.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:37.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:37.352 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:37.394 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:37.394 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:37.655 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.19.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:36:37.656 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:36:37.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:37.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:37.658 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:37.703 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:37.703 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:37.964 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:36:37.964 WARNING:rgw_multi.tests:bucket pdivcf-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.19.12 target=00000000001.16.5 2020-07-23T12:36:42.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:42.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:42.967 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:43.012 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:43.013 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:43.278 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:36:43.279 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:36:43.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:43.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:43.281 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:43.325 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:43.325 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:43.574 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.19.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:36:43.575 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:36:43.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:43.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:43.577 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:43.659 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:43.659 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:43.934 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:36:43.935 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:36:43.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:43.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:43.938 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:44.023 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:44.023 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:44.231 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.19.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:36:44.232 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:36:44.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:44.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:44.235 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:44.287 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:44.287 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:44.562 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.19.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:36:44.563 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:36:44.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:44.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:44.566 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:44.646 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:44.646 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:44.878 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.20.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:36:44.879 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-20 2020-07-23T12:36:44.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:44.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:44.880 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:44.929 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:44.929 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:45.196 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.20.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:36:45.196 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-20 2020-07-23T12:36:45.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:45.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:45.198 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:45.244 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:45.244 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:45.475 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.20.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:36:45.476 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-20 2020-07-23T12:36:45.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:45.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:45.479 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:45.522 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:45.522 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:45.778 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.20.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:36:45.778 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-20 2020-07-23T12:36:45.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:45.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:45.780 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:45.870 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:45.870 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:46.089 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.22.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:36:46.090 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-20 2020-07-23T12:36:46.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:46.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:46.093 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:46.175 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:46.175 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:46.438 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:36:46.438 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-20 2020-07-23T12:36:46.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:46.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:46.440 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:46.520 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:46.520 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:46.759 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.22.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:36:46.759 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-20 2020-07-23T12:36:46.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:46.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:46.761 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:46.806 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:46.807 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:47.082 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:36:47.083 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-20 2020-07-23T12:36:47.084 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-20 zones={test-zone2, test-zone1} 2020-07-23T12:36:48.326 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:36:48.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:48.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:48.337 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:36:48.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:48.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:48.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:48.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:48.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:48.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:48.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:48.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:48.766 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-20 zones={test-zone2, test-zone1} 2020-07-23T12:36:48.766 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-20 zones={test-zone3, test-zone1} 2020-07-23T12:36:49.521 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:36:49.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:49.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:49.571 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:36:49.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:49.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:49.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:49.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:49.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:49.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:49.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:49.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:49.946 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-20 zones={test-zone3, test-zone1} 2020-07-23T12:36:49.947 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-20 zones={test-zone3, test-zone2} 2020-07-23T12:36:50.495 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:36:50.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:50.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:50.507 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:36:50.558 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:50.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:36:50.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:50.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:50.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:50.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:50.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:50.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:36:50.969 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-20 zones={test-zone3, test-zone2} 2020-07-23T12:36:51.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:51.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:51.971 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:52.012 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:52.012 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:52.223 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:36:52.262 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595507528.176421_165.1","last_update":"2020-07-23T12:32:08.176421Z"},{"marker":"1_1595507811.963376_225.1","last_update":"2020-07-23T12:36:51.963376Z"}] 2020-07-23T12:36:52.263 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:36:52.264 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:52.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:52.265 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:52.304 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:52.305 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:52.569 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507790.379387_217.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:36:30.379387Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:36:52.570 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595507811.963376_225.1 target=1_1595507790.379387_217.1 2020-07-23T12:36:57.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:57.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:57.575 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:57.617 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:57.618 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:57.860 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507811.963376_225.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:36:51.963376Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:36:57.861 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:36:57.862 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:36:57.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:57.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:57.865 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:57.908 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:57.908 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:58.148 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507811.963376_225.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:36:51.963376Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:36:58.149 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:36:58.149 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:36:58.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:58.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:58.151 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:58.192 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:58.193 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:58.463 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507811.963376_225.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:36:51.963376Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:36:58.463 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:36:59.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:59.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:59.108 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:59.149 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:59.150 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:59.386 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.23.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:36:59.387 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:36:59.387 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:59.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:36:59.388 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:36:59.445 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:36:59.445 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:36:59.695 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:36:59.695 WARNING:rgw_multi.tests:bucket pdivcf-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000005.23.17 target=00000000003.19.12 2020-07-23T12:37:04.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:04.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:04.699 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:04.760 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:04.761 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:05.008 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.23.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:05.009 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:37:05.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:05.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:05.011 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:05.051 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:05.052 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:05.285 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.23.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:05.286 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:37:05.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:05.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:05.288 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:05.373 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:05.374 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:05.644 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.23.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:05.645 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:37:05.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:05.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:05.648 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:05.727 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:05.727 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:05.982 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.23.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:05.983 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:37:05.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:05.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:05.985 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:06.032 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:06.032 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:06.307 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.23.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:06.308 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-20 2020-07-23T12:37:06.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:06.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:06.312 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:06.397 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:06.397 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:06.642 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.28.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:06.642 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-20 2020-07-23T12:37:06.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:06.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:06.645 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:06.690 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:06.690 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:06.953 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.28.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:06.954 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-20 2020-07-23T12:37:06.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:06.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:06.974 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:07.015 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:07.016 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:07.265 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.28.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:07.265 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-20 2020-07-23T12:37:07.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:07.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:07.274 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:07.340 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:07.342 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:07.610 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.28.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:07.611 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-20 2020-07-23T12:37:07.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:07.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:07.613 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:07.690 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:07.691 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:07.939 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.30.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:07.939 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-20 2020-07-23T12:37:07.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:07.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:07.941 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:08.024 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:08.025 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:08.296 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.30.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:08.296 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-20 2020-07-23T12:37:08.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:08.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:08.298 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:08.384 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:08.386 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:08.623 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.30.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:08.623 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-20 2020-07-23T12:37:08.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:08.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:08.625 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:08.667 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:08.667 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:08.915 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.30.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:08.916 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-20 2020-07-23T12:37:08.916 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-20 zones={test-zone2, test-zone1} 2020-07-23T12:37:09.946 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:09.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:09.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:09.959 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:10.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:10.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:10.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:10.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:10.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:10.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:10.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:10.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:10.369 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-20 zones={test-zone2, test-zone1} 2020-07-23T12:37:10.370 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-20 zones={test-zone3, test-zone1} 2020-07-23T12:37:11.369 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:11.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:11.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:11.419 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:11.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:11.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:11.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:11.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:11.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:11.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:11.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:11.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:11.781 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-20 zones={test-zone3, test-zone1} 2020-07-23T12:37:11.783 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-20 zones={test-zone3, test-zone2} 2020-07-23T12:37:12.704 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:12.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:12.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:12.733 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:12.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:12.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:12.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:12.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:12.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:13.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:13.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:13.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:13.190 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-20 zones={test-zone3, test-zone2} 2020-07-23T12:37:13.191 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2020-07-23T12:37:13.193 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-21 2020-07-23T12:37:13.996 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-22 2020-07-23T12:37:14.055 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-23 2020-07-23T12:37:14.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:14.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:14.152 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:14.184 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:14.185 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:14.373 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:37:14.410 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595507528.176421_165.1","last_update":"2020-07-23T12:32:08.176421Z"},{"marker":"1_1595507834.142058_249.1","last_update":"2020-07-23T12:37:14.142058Z"}] 2020-07-23T12:37:14.411 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:37:14.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:14.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:14.413 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:14.458 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:14.459 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:14.698 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507811.963376_225.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:36:51.963376Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:37:14.699 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595507834.142058_249.1 target=1_1595507811.963376_225.1 2020-07-23T12:37:19.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:19.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:19.702 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:19.743 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:19.743 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:19.987 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507834.142058_249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:37:14.142058Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:37:19.988 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:37:19.988 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:37:19.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:19.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:19.990 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:20.031 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:20.032 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:20.258 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507834.142058_249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:37:14.142058Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:37:20.258 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:37:20.259 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:37:20.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:20.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:20.261 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:20.301 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:20.302 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:20.574 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507834.142058_249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:37:14.142058Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:37:20.575 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:37:22.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:22.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:22.435 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:22.476 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:22.477 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:22.750 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.28.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:22.751 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-21 2020-07-23T12:37:22.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:22.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:22.752 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:22.797 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:22.798 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:23.047 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zZqqwX6l0URm0btqf6sbKRd5OSQXFqK","ns":""},"count":1},"inc_marker":{"position":"00000000003.28.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:23.048 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-21 2020-07-23T12:37:23.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:23.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:23.050 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:23.098 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:23.099 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:23.319 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.28.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:23.320 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-21 2020-07-23T12:37:23.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:23.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:23.323 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:23.369 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:23.369 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:23.617 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zZqqwX6l0URm0btqf6sbKRd5OSQXFqK","ns":""},"count":1},"inc_marker":{"position":"00000000003.28.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:23.618 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-21 2020-07-23T12:37:23.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:23.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:23.619 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:23.697 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:23.697 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:23.939 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.28.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:23.939 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-21 2020-07-23T12:37:23.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:23.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:23.941 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:23.988 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:23.988 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:24.243 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.28.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:24.244 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-21 2020-07-23T12:37:24.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:24.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:24.246 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:24.285 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:24.286 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:24.518 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.33.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:24.518 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-21 2020-07-23T12:37:24.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:24.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:24.520 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:24.575 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:24.576 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:24.832 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zZqqwX6l0URm0btqf6sbKRd5OSQXFqK","ns":""},"count":1},"inc_marker":{"position":"00000000002.33.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:24.833 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-21 2020-07-23T12:37:24.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:24.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:24.835 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:24.872 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:24.873 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:25.110 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.33.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:25.111 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-21 2020-07-23T12:37:25.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:25.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:25.112 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:25.156 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:25.156 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:25.437 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zZqqwX6l0URm0btqf6sbKRd5OSQXFqK","ns":""},"count":1},"inc_marker":{"position":"00000000002.33.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:25.438 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-21 2020-07-23T12:37:25.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:25.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:25.440 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:25.520 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:25.520 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:25.748 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.33.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:25.749 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-21 2020-07-23T12:37:25.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:25.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:25.751 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:25.837 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:25.838 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:26.104 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zZqqwX6l0URm0btqf6sbKRd5OSQXFqK","ns":""},"count":1},"inc_marker":{"position":"00000000002.33.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:26.105 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-21 2020-07-23T12:37:26.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:26.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:26.107 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:26.190 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:26.190 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:26.448 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.33.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:26.448 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-21 2020-07-23T12:37:26.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:26.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:26.450 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:26.494 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:26.494 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:26.742 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zZqqwX6l0URm0btqf6sbKRd5OSQXFqK","ns":""},"count":1},"inc_marker":{"position":"00000000002.33.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:26.743 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-21 2020-07-23T12:37:26.744 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-21 zones={test-zone2, test-zone1} 2020-07-23T12:37:27.479 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:27.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:27.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:27.495 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:27.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:27.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:27.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:27.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:27.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:27.716 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-21 zones={test-zone2, test-zone1} 2020-07-23T12:37:27.717 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-21 zones={test-zone3, test-zone1} 2020-07-23T12:37:28.325 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:28.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:28.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:28.376 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:28.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:28.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:28.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:28.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:28.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:28.585 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-21 zones={test-zone3, test-zone1} 2020-07-23T12:37:28.586 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-21 zones={test-zone3, test-zone2} 2020-07-23T12:37:28.596 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:28.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:28.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:28.644 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:28.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:28.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:28.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:28.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:28.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:28.803 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-21 zones={test-zone3, test-zone2} 2020-07-23T12:37:28.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:28.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:28.806 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:28.854 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:28.854 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:29.090 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.14.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:29.091 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-22 2020-07-23T12:37:29.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:29.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:29.093 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:29.160 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:29.160 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:29.405 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Yd9j87einwJepRhmeI8lsMiuMN1FQkY","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:29.406 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-22 2020-07-23T12:37:29.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:29.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:29.408 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:29.457 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:29.458 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:29.709 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.14.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:29.710 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-22 2020-07-23T12:37:29.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:29.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:29.712 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:29.795 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:29.795 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:30.061 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Yd9j87einwJepRhmeI8lsMiuMN1FQkY","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:30.062 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-22 2020-07-23T12:37:30.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:30.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:30.067 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:30.144 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:30.144 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:30.379 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.14.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:30.380 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-22 2020-07-23T12:37:30.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:30.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:30.396 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:30.441 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:30.443 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:30.705 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.14.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:30.706 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-22 2020-07-23T12:37:30.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:30.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:30.708 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:30.750 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:30.750 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:30.979 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.16.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:30.979 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-22 2020-07-23T12:37:30.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:30.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:30.981 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:31.030 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:31.031 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:31.302 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Yd9j87einwJepRhmeI8lsMiuMN1FQkY","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:31.303 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-22 2020-07-23T12:37:31.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:31.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:31.305 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:31.348 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:31.349 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:31.570 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.16.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:31.571 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-22 2020-07-23T12:37:31.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:31.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:31.573 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:31.626 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:31.626 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:31.908 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Yd9j87einwJepRhmeI8lsMiuMN1FQkY","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:31.908 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-22 2020-07-23T12:37:31.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:31.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:31.910 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:31.985 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:31.986 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:32.197 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.17.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:32.197 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-22 2020-07-23T12:37:32.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:32.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:32.199 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:32.283 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:32.283 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:32.570 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Yd9j87einwJepRhmeI8lsMiuMN1FQkY","ns":""},"count":1},"inc_marker":{"position":"00000000002.17.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:32.571 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-22 2020-07-23T12:37:32.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:32.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:32.573 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:32.653 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:32.653 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:32.892 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.17.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:32.893 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-22 2020-07-23T12:37:32.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:32.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:32.894 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:32.938 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:32.939 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:33.204 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Yd9j87einwJepRhmeI8lsMiuMN1FQkY","ns":""},"count":1},"inc_marker":{"position":"00000000002.17.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:33.205 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-22 2020-07-23T12:37:33.205 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-22 zones={test-zone2, test-zone1} 2020-07-23T12:37:34.237 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:34.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:34.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:34.246 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:34.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:34.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:34.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:34.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:34.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:34.466 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-22 zones={test-zone2, test-zone1} 2020-07-23T12:37:34.468 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-22 zones={test-zone3, test-zone1} 2020-07-23T12:37:35.149 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:35.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:35.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:35.200 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:35.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:35.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:35.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:35.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:35.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:35.359 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-22 zones={test-zone3, test-zone1} 2020-07-23T12:37:35.360 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-22 zones={test-zone3, test-zone2} 2020-07-23T12:37:35.382 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:35.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:35.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:35.443 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:35.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:35.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:35.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:35.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:35.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:35.627 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-22 zones={test-zone3, test-zone2} 2020-07-23T12:37:35.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:35.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:35.631 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:35.678 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:35.678 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:35.942 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.21.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:35.943 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-23 2020-07-23T12:37:35.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:35.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:35.944 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:36.003 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:36.004 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:36.260 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"l7wCX3yxSmTSFG566Yb4ufRq1lmYlFC","ns":""},"count":1},"inc_marker":{"position":"00000000002.21.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:36.261 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-23 2020-07-23T12:37:36.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:36.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:36.262 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:36.302 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:36.302 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:36.565 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.21.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:36.566 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-23 2020-07-23T12:37:36.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:36.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:36.568 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:36.658 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:36.659 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:36.947 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"l7wCX3yxSmTSFG566Yb4ufRq1lmYlFC","ns":""},"count":1},"inc_marker":{"position":"00000000002.21.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:36.947 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-23 2020-07-23T12:37:36.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:36.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:36.950 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:37.025 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:37.026 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:37.270 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.21.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:37.272 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-23 2020-07-23T12:37:37.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:37.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:37.278 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:37.321 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:37.322 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:37.596 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:37.597 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-23 2020-07-23T12:37:37.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:37.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:37.606 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:37.680 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:37.681 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:37.920 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.26.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:37.921 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-23 2020-07-23T12:37:37.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:37.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:37.936 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:37.984 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:37.985 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:38.264 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"l7wCX3yxSmTSFG566Yb4ufRq1lmYlFC","ns":""},"count":1},"inc_marker":{"position":"00000000002.26.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:38.266 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-23 2020-07-23T12:37:38.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:38.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:38.267 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:38.308 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:38.308 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:38.548 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.26.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:38.548 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-23 2020-07-23T12:37:38.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:38.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:38.551 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:38.595 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:38.596 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:38.851 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"l7wCX3yxSmTSFG566Yb4ufRq1lmYlFC","ns":""},"count":1},"inc_marker":{"position":"00000000002.26.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:38.852 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-23 2020-07-23T12:37:38.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:38.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:38.853 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:38.937 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:38.938 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:39.176 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.28.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:39.177 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-23 2020-07-23T12:37:39.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:39.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:39.179 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:39.264 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:39.265 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:39.532 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"l7wCX3yxSmTSFG566Yb4ufRq1lmYlFC","ns":""},"count":1},"inc_marker":{"position":"00000000003.28.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:39.533 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-23 2020-07-23T12:37:39.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:39.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:39.535 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:39.575 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:39.576 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:39.832 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.28.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:39.833 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-23 2020-07-23T12:37:39.833 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:39.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:39.835 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:39.886 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:39.887 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:40.152 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"l7wCX3yxSmTSFG566Yb4ufRq1lmYlFC","ns":""},"count":1},"inc_marker":{"position":"00000000003.28.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:40.153 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-23 2020-07-23T12:37:40.154 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-23 zones={test-zone2, test-zone1} 2020-07-23T12:37:41.426 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:41.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:41.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:41.442 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:41.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:41.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:41.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:41.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:41.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:41.660 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-23 zones={test-zone2, test-zone1} 2020-07-23T12:37:41.661 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-23 zones={test-zone3, test-zone1} 2020-07-23T12:37:41.751 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:41.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:41.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:41.804 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:41.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:41.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:41.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:41.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:41.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:42.017 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-23 zones={test-zone3, test-zone1} 2020-07-23T12:37:42.018 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-23 zones={test-zone3, test-zone2} 2020-07-23T12:37:42.029 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:42.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:42.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:42.080 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:42.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:42.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:42.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:42.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:42.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:42.278 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-23 zones={test-zone3, test-zone2} 2020-07-23T12:37:42.279 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2020-07-23T12:37:42.280 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-24 2020-07-23T12:37:42.319 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-25 2020-07-23T12:37:42.377 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-26 2020-07-23T12:37:42.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:42.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:42.579 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:42.613 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:42.613 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:42.835 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:37:42.870 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595507528.176421_165.1","last_update":"2020-07-23T12:32:08.176421Z"},{"marker":"1_1595507862.561233_297.1","last_update":"2020-07-23T12:37:42.561233Z"}] 2020-07-23T12:37:42.871 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:37:42.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:42.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:42.873 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:42.912 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:42.912 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:43.139 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507862.561233_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:37:42.561233Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:37:43.139 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:37:43.140 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:37:43.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:43.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:43.142 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:43.182 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:43.183 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:43.407 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507862.561233_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:37:42.561233Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:37:43.408 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:37:43.408 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:37:43.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:43.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:43.410 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:43.451 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:43.451 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:43.719 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507862.561233_297.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:37:42.561233Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:37:43.720 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:37:44.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:44.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:44.932 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:44.974 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:44.975 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:45.219 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.12.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:45.219 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-24 2020-07-23T12:37:45.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:45.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:45.221 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:45.267 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:45.268 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:45.503 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.12.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:45.504 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-24 2020-07-23T12:37:45.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:45.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:45.507 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:45.549 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:45.549 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:45.782 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.12.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:45.783 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-24 2020-07-23T12:37:45.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:45.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:45.785 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:45.872 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:45.873 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:46.097 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.12.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:46.098 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-24 2020-07-23T12:37:46.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:46.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:46.100 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:46.181 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:46.182 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:46.432 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.12.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:46.432 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-24 2020-07-23T12:37:46.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:46.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:46.435 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:46.478 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:46.479 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:46.746 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.12.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:46.746 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-24 2020-07-23T12:37:46.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:46.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:46.748 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:46.829 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:46.830 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:47.089 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.13.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:47.090 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-24 2020-07-23T12:37:47.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:47.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:47.092 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:47.136 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:47.137 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:47.418 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:47.418 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-24 2020-07-23T12:37:47.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:47.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:47.420 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:47.468 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:47.468 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:47.674 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.13.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:47.674 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-24 2020-07-23T12:37:47.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:47.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:47.676 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:47.721 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:47.721 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:47.998 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:47.999 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-24 2020-07-23T12:37:48.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:48.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:48.001 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:48.078 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:48.079 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:48.293 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.9.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:48.294 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-24 2020-07-23T12:37:48.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:48.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:48.296 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:48.376 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:48.377 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:48.656 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:48.657 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-24 2020-07-23T12:37:48.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:48.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:48.659 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:48.739 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:48.739 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:48.995 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.9.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:48.996 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-24 2020-07-23T12:37:48.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:48.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:48.997 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:49.051 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:49.052 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:49.317 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:49.317 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-24 2020-07-23T12:37:49.318 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-24 zones={test-zone2, test-zone1} 2020-07-23T12:37:49.399 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:49.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:49.409 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:49.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:49.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:49.532 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-24 zones={test-zone2, test-zone1} 2020-07-23T12:37:49.540 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-24 zones={test-zone3, test-zone1} 2020-07-23T12:37:50.239 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:50.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:50.287 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:50.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:50.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:50.362 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-24 zones={test-zone3, test-zone1} 2020-07-23T12:37:50.363 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-24 zones={test-zone3, test-zone2} 2020-07-23T12:37:50.375 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:50.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:50.423 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:50.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:50.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:50.503 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-24 zones={test-zone3, test-zone2} 2020-07-23T12:37:50.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:50.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:50.505 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:50.555 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:50.555 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:50.806 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.16.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:50.807 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-25 2020-07-23T12:37:50.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:50.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:50.809 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:50.858 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:50.859 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:51.118 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:51.119 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-25 2020-07-23T12:37:51.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:51.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:51.122 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:51.163 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:51.164 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:51.428 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.16.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:51.429 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-25 2020-07-23T12:37:51.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:51.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:51.433 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:51.514 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:51.514 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:51.769 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:51.770 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-25 2020-07-23T12:37:51.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:51.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:51.773 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:51.849 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:51.850 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:52.082 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.16.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:52.083 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-25 2020-07-23T12:37:52.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:52.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:52.085 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:52.129 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:52.130 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:52.404 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.16.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:52.405 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-25 2020-07-23T12:37:52.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:52.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:52.407 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:52.453 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:52.454 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:52.683 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.26.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:52.683 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-25 2020-07-23T12:37:52.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:52.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:52.685 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:52.728 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:52.729 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:52.988 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.26.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:52.988 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-25 2020-07-23T12:37:52.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:52.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:52.991 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:53.031 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:53.032 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:53.287 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.26.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:53.288 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-25 2020-07-23T12:37:53.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:53.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:53.290 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:53.337 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:53.338 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:53.597 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.26.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:53.598 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-25 2020-07-23T12:37:53.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:53.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:53.601 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:53.641 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:53.641 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:53.888 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.18.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:53.889 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-25 2020-07-23T12:37:53.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:53.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:53.891 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:53.973 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:53.973 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:54.238 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.18.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:54.239 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-25 2020-07-23T12:37:54.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:54.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:54.241 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:54.328 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:54.328 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:54.579 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.18.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:54.580 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-25 2020-07-23T12:37:54.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:54.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:54.582 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:54.629 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:54.629 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:54.861 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.18.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:54.862 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-25 2020-07-23T12:37:54.862 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-25 zones={test-zone2, test-zone1} 2020-07-23T12:37:56.151 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:56.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:56.160 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:56.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:56.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:56.277 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-25 zones={test-zone2, test-zone1} 2020-07-23T12:37:56.279 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-25 zones={test-zone3, test-zone1} 2020-07-23T12:37:56.947 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:57.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:57.007 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:57.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:57.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:57.076 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-25 zones={test-zone3, test-zone1} 2020-07-23T12:37:57.077 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-25 zones={test-zone3, test-zone2} 2020-07-23T12:37:57.088 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:37:57.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:57.135 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:37:57.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:37:57.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:37:57.206 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-25 zones={test-zone3, test-zone2} 2020-07-23T12:37:57.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:57.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:57.209 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:57.251 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:57.251 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:57.504 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.17.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:57.504 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-26 2020-07-23T12:37:57.505 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:57.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:57.507 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:57.552 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:57.553 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:57.817 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:57.818 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-26 2020-07-23T12:37:57.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:57.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:57.821 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:57.864 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:57.865 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:58.099 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.17.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:58.100 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-26 2020-07-23T12:37:58.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:58.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:58.102 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:58.185 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:58.186 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:58.437 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:58.438 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-26 2020-07-23T12:37:58.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:58.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:58.439 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:58.521 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:58.522 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:58.764 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.17.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:58.764 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-26 2020-07-23T12:37:58.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:58.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:58.766 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:58.818 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:58.818 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:59.080 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.17.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:59.081 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-26 2020-07-23T12:37:59.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:59.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:59.083 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:59.160 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:59.160 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:59.400 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.17.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:37:59.401 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-26 2020-07-23T12:37:59.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:59.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:59.403 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:59.447 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:59.447 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:37:59.713 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:37:59.714 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-26 2020-07-23T12:37:59.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:59.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:37:59.717 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:37:59.767 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:37:59.768 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:38:00.007 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.17.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:38:00.008 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-26 2020-07-23T12:38:00.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:00.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:00.011 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:38:00.053 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:38:00.053 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:38:00.308 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:38:00.310 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-26 2020-07-23T12:38:00.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:00.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:00.313 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:38:00.393 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:38:00.393 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:38:00.640 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.25.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:38:00.641 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-26 2020-07-23T12:38:00.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:00.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:00.644 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:38:00.729 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:38:00.729 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:38:01.004 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.25.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:38:01.004 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-26 2020-07-23T12:38:01.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:01.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:01.006 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:38:01.090 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:38:01.090 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:38:01.337 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.25.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:38:01.338 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-26 2020-07-23T12:38:01.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:01.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:01.340 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:38:01.386 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:38:01.387 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:38:01.626 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.25.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:38:01.626 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-26 2020-07-23T12:38:01.627 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-26 zones={test-zone2, test-zone1} 2020-07-23T12:38:01.803 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:38:01.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:38:01.855 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:38:01.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:38:01.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:38:01.927 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-26 zones={test-zone2, test-zone1} 2020-07-23T12:38:01.928 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-26 zones={test-zone3, test-zone1} 2020-07-23T12:38:02.438 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:38:02.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:38:02.487 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:38:02.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:38:02.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:38:02.561 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-26 zones={test-zone3, test-zone1} 2020-07-23T12:38:02.562 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-26 zones={test-zone3, test-zone2} 2020-07-23T12:38:02.573 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:38:02.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:38:02.627 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:38:02.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-07-23T12:38:02.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-07-23T12:38:02.698 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-26 zones={test-zone3, test-zone2} 2020-07-23T12:38:02.699 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2020-07-23T12:38:02.700 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-27 2020-07-23T12:38:02.738 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-28 2020-07-23T12:38:02.804 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-29 2020-07-23T12:38:03.062 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2020-07-23T12:38:03.063 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-30 2020-07-23T12:38:03.106 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-31 2020-07-23T12:38:03.158 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-32 2020-07-23T12:38:03.412 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2020-07-23T12:38:03.413 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-33 2020-07-23T12:38:03.470 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-34 2020-07-23T12:38:03.595 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-35 2020-07-23T12:38:05.934 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2020-07-23T12:38:05.935 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-36 2020-07-23T12:38:06.317 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-37 2020-07-23T12:38:07.167 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-38 2020-07-23T12:38:07.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:07.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:07.557 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:38:07.594 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:38:07.594 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:38:07.815 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:38:07.854 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595507528.176421_165.1","last_update":"2020-07-23T12:32:08.176421Z"},{"marker":"1_1595507887.526314_381.1","last_update":"2020-07-23T12:38:07.526314Z"}] 2020-07-23T12:38:07.855 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:38:07.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:07.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:07.857 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:38:07.897 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:38:07.898 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:38:08.132 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507887.526314_381.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:07.526314Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:38:08.132 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:38:08.133 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:38:08.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:08.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:08.134 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:38:08.176 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:38:08.176 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:38:08.427 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507887.526314_381.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:07.526314Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:38:08.428 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:38:08.429 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:38:08.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:08.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:08.430 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:38:08.470 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:38:08.470 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:38:08.751 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507887.526314_381.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:07.526314Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:38:08.752 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:38:10.066 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2020-07-23T12:38:10.068 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-39 2020-07-23T12:38:10.102 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-40 2020-07-23T12:38:10.175 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-41 2020-07-23T12:38:10.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:10.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:10.253 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:38:10.291 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:38:10.291 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:38:10.501 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=03aec9f8-1b80-4c7c-ae43-862168b0b5e6 2020-07-23T12:38:10.539 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507375.329399_72.1","last_update":"2020-07-23T12:29:35.329399Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595507528.176421_165.1","last_update":"2020-07-23T12:32:08.176421Z"},{"marker":"1_1595507890.217061_394.1","last_update":"2020-07-23T12:38:10.217061Z"}] 2020-07-23T12:38:10.540 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:38:10.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:10.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:10.542 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:38:10.588 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:38:10.589 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:38:10.818 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507887.526314_381.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:07.526314Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:38:10.819 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595507890.217061_394.1 target=1_1595507887.526314_381.1 2020-07-23T12:38:15.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:15.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:15.823 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:38:15.864 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:38:15.865 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:38:16.107 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507890.217061_394.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:10.217061Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:38:16.107 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:38:16.108 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:38:16.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:16.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:16.110 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:38:16.151 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:38:16.151 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:38:16.389 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507890.217061_394.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:10.217061Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:38:16.389 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:38:16.390 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:38:16.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:16.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:38:16.392 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:38:16.433 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:38:16.434 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:38:16.709 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595507890.217061_394.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:10.217061Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:38:16.709 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:38:16.710 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-07-23T12:38:16.711 INFO:teuthology.orchestra.run:waiting for 300 2020-07-23T12:38:16.712 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:38:16.708+0000 7f315affd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 16231) UID: 0 2020-07-23T12:38:16.712 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:38:16.708+0000 7f3176ced0c0 -1 shutting down 2020-07-23T12:38:19.299 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-07-23T12:38:22.813 INFO:tasks.rgw.c1.client.1:Stopped 2020-07-23T12:38:22.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:38:22.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:38:22.815 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:38:22.854 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:38:22.854 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:38:22.880 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:38:22.875+0000 7fb518ed3080 0 NOTICE: overriding master zone: 0378667f-02f5-4ce6-824e-e2a396c12fc3 2020-07-23T12:38:22.881 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:38:22.875+0000 7fb518ed3080 0 failed reading obj info from .rgw.root:zone_info.0378667f-02f5-4ce6-824e-e2a396c12fc3: (2) No such file or directory 2020-07-23T12:38:22.882 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:38:22.875+0000 7fb518ed3080 0 WARNING: could not read zone params for zone id=0378667f-02f5-4ce6-824e-e2a396c12fc3 name=test-zone1 2020-07-23T12:38:22.882 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:38:22.879+0000 7fb518ed3080 0 failed reading obj info from .rgw.root:zone_info.41767e64-47c7-45a2-9984-76631257f2f6: (2) No such file or directory 2020-07-23T12:38:22.883 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:38:22.879+0000 7fb518ed3080 0 WARNING: could not read zone params for zone id=41767e64-47c7-45a2-9984-76631257f2f6 name=test-zone3 2020-07-23T12:38:22.898 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"T855BRGPSH14I4Y2","secret_key":"mMgfonT3dSbKAOArpyZfLcrPB3lPybUX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca"} 2020-07-23T12:38:22.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:38:22.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:38:22.901 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:38:22.980 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:38:22.980 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:38:23.299 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"30fc0b4a-5063-4d9f-a69b-5de0a471142a","epoch":1,"predecessor_uuid":"03aec9f8-1b80-4c7c-ae43-862168b0b5e6","sync_status":["1_1595507375.329399_72.1","","1_1595507528.176421_165.1","1_1595507890.217061_394.1"],"period_map":{"id":"30fc0b4a-5063-4d9f-a69b-5de0a471142a","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"42997773-3698-4448-bf7b-a00a5846d584","zones":[{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"32a3ec76-95f5-4863-891c-667a67c89f48","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"41767e64-47c7-45a2-9984-76631257f2f6","name":"test-zone3","endpoints":["http://smithi137.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0378667f-02f5-4ce6-824e-e2a396c12fc3","val":68328169},{"key":"32a3ec76-95f5-4863-891c-667a67c89f48","val":3058828581},{"key":"41767e64-47c7-45a2-9984-76631257f2f6","val":2499065231},{"key":"42997773-3698-4448-bf7b-a00a5846d584","val":1484105665}]},"master_zonegroup":"test-zonegroup","master_zone":"42997773-3698-4448-bf7b-a00a5846d584","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","realm_name":"test-realm","realm_epoch":3} 2020-07-23T12:38:23.300 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2020-07-23T12:38:24.904 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-07-23T12:39:23.302 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-42 2020-07-23T12:39:23.383 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-43 2020-07-23T12:39:23.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:39:23.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:39:23.427 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:39:23.469 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:39:23.469 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:39:23.643 INFO:teuthology.orchestra.run.smithi133.stderr:No --period given, using current period=30fc0b4a-5063-4d9f-a69b-5de0a471142a 2020-07-23T12:39:23.678 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595507963.420183_265.1","last_update":"2020-07-23T12:39:23.420183Z"}] 2020-07-23T12:39:23.679 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2020-07-23T12:39:23.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:39:23.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:39:23.681 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:39:23.720 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:39:23.720 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:39:23.991 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"30fc0b4a-5063-4d9f-a69b-5de0a471142a","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":80,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":80,"complete":80}} 2020-07-23T12:39:23.992 WARNING:rgw_multi.tests:zone test-zone1 behind master: shard=3 master=1_1595507963.420183_265.1 target= 2020-07-23T12:39:28.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:39:28.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:39:28.998 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:39:29.042 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:39:29.042 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:39:29.303 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"30fc0b4a-5063-4d9f-a69b-5de0a471142a","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":80,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"1_1595507963.420183_265.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:39:23.420183Z","realm_epoch":3}}]},"full_sync":{"total":80,"complete":80}} 2020-07-23T12:39:29.303 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2020-07-23T12:39:29.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:39:29.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:39:29.307 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:39:29.390 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:39:29.391 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:39:29.419 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:39:29.411+0000 7f02d98df080 0 NOTICE: overriding master zone: 42997773-3698-4448-bf7b-a00a5846d584 2020-07-23T12:39:29.420 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:39:29.415+0000 7f02d98df080 0 failed reading obj info from .rgw.root:zone_info.32a3ec76-95f5-4863-891c-667a67c89f48: (2) No such file or directory 2020-07-23T12:39:29.420 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:39:29.415+0000 7f02d98df080 0 WARNING: could not read zone params for zone id=32a3ec76-95f5-4863-891c-667a67c89f48 name=test-zone4 2020-07-23T12:39:29.425 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:39:29.415+0000 7f02d98df080 0 failed reading obj info from .rgw.root:zone_info.42997773-3698-4448-bf7b-a00a5846d584: (2) No such file or directory 2020-07-23T12:39:29.425 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:39:29.415+0000 7f02d98df080 0 WARNING: could not read zone params for zone id=42997773-3698-4448-bf7b-a00a5846d584 name=test-zone2 2020-07-23T12:39:29.432 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"T855BRGPSH14I4Y2","secret_key":"mMgfonT3dSbKAOArpyZfLcrPB3lPybUX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca"} 2020-07-23T12:39:29.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:39:29.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:39:29.434 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:39:29.516 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:39:29.516 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:39:29.866 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","epoch":1,"predecessor_uuid":"30fc0b4a-5063-4d9f-a69b-5de0a471142a","sync_status":["","","","1_1595507963.420183_265.1"],"period_map":{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","zones":[{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"32a3ec76-95f5-4863-891c-667a67c89f48","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"41767e64-47c7-45a2-9984-76631257f2f6","name":"test-zone3","endpoints":["http://smithi137.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0378667f-02f5-4ce6-824e-e2a396c12fc3","val":68328169},{"key":"32a3ec76-95f5-4863-891c-667a67c89f48","val":3058828581},{"key":"41767e64-47c7-45a2-9984-76631257f2f6","val":2499065231},{"key":"42997773-3698-4448-bf7b-a00a5846d584","val":1484105665}]},"master_zonegroup":"test-zonegroup","master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","realm_name":"test-realm","realm_epoch":4} 2020-07-23T12:39:29.867 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2020-07-23T12:40:29.869 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-44 2020-07-23T12:40:29.926 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-45 2020-07-23T12:40:30.005 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-07-23T12:40:30.007 INFO:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-07-23T12:40:30.012 INFO:tasks.rgw.c1.client.1:Started 2020-07-23T12:40:30.013 INFO:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8001 2020-07-23T12:40:30.029 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-23T12:40:30.029 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2020-07-23T12:40:30.033 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi137.front.sepia.ceph.com port 8001: Connection refused 2020-07-23T12:40:30.033 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-07-23T12:40:31.034 INFO:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8001 2020-07-23T12:40:31.049 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-07-23T12:40:31.050 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2020-07-23T12:40:31.054 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2020-07-23T12:40:31.055 INFO:teuthology.orchestra.run.smithi137.stdout:anonymous 2020-07-23T12:40:31.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:31.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:31.057 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:31.145 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:31.145 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:31.357 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=a2b03b35-661e-4182-91b6-d60b3b65f694 2020-07-23T12:40:31.397 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595508029.980256_108.1","last_update":"2020-07-23T12:40:29.980256Z"}] 2020-07-23T12:40:31.398 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:40:31.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:31.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:31.400 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:31.442 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:31.442 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:31.672 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508029.980256_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:29.980256Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:40:31.673 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:40:31.674 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:40:31.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:31.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:31.675 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:31.717 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:31.717 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:31.985 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508029.980256_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:29.980256Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:40:31.986 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:40:31.987 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:40:31.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:31.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:31.988 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:32.027 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:32.027 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:32.295 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508029.980256_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:29.980256Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:40:32.296 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:40:32.296 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-39 zones={test-zone2, test-zone1} 2020-07-23T12:40:33.886 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:33.894 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:33.998 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-39 zones={test-zone2, test-zone1} 2020-07-23T12:40:33.999 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-39 zones={test-zone3, test-zone1} 2020-07-23T12:40:34.022 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:34.070 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:34.134 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-39 zones={test-zone3, test-zone1} 2020-07-23T12:40:34.135 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-39 zones={test-zone1, test-zone2} 2020-07-23T12:40:34.148 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:34.198 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:34.264 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-39 zones={test-zone1, test-zone2} 2020-07-23T12:40:34.265 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-39 zones={test-zone3, test-zone2} 2020-07-23T12:40:34.277 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:34.326 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:34.395 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-39 zones={test-zone3, test-zone2} 2020-07-23T12:40:34.396 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-39 zones={test-zone1, test-zone3} 2020-07-23T12:40:34.409 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:34.458 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:34.526 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-39 zones={test-zone1, test-zone3} 2020-07-23T12:40:34.527 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-39 zones={test-zone2, test-zone3} 2020-07-23T12:40:34.537 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:34.594 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:34.658 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-39 zones={test-zone2, test-zone3} 2020-07-23T12:40:34.659 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-40 zones={test-zone2, test-zone1} 2020-07-23T12:40:34.673 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:34.722 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:34.790 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-40 zones={test-zone2, test-zone1} 2020-07-23T12:40:34.791 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-40 zones={test-zone3, test-zone1} 2020-07-23T12:40:34.801 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:34.850 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:34.913 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-40 zones={test-zone3, test-zone1} 2020-07-23T12:40:34.914 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-40 zones={test-zone1, test-zone2} 2020-07-23T12:40:34.925 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:34.970 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:35.038 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-40 zones={test-zone1, test-zone2} 2020-07-23T12:40:35.039 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-40 zones={test-zone3, test-zone2} 2020-07-23T12:40:35.048 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:35.098 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:35.164 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-40 zones={test-zone3, test-zone2} 2020-07-23T12:40:35.165 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-40 zones={test-zone1, test-zone3} 2020-07-23T12:40:35.178 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:35.226 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:35.293 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-40 zones={test-zone1, test-zone3} 2020-07-23T12:40:35.294 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-40 zones={test-zone2, test-zone3} 2020-07-23T12:40:35.304 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:35.354 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:35.420 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-40 zones={test-zone2, test-zone3} 2020-07-23T12:40:35.421 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-41 zones={test-zone2, test-zone1} 2020-07-23T12:40:35.434 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:35.482 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:35.549 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-41 zones={test-zone2, test-zone1} 2020-07-23T12:40:35.550 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-41 zones={test-zone3, test-zone1} 2020-07-23T12:40:35.561 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:35.610 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:35.672 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-41 zones={test-zone3, test-zone1} 2020-07-23T12:40:35.673 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-41 zones={test-zone1, test-zone2} 2020-07-23T12:40:35.684 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:35.730 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:35.793 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-41 zones={test-zone1, test-zone2} 2020-07-23T12:40:35.794 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-41 zones={test-zone3, test-zone2} 2020-07-23T12:40:35.803 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:35.850 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:35.913 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-41 zones={test-zone3, test-zone2} 2020-07-23T12:40:35.914 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-41 zones={test-zone1, test-zone3} 2020-07-23T12:40:35.926 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:35.974 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:36.038 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-41 zones={test-zone1, test-zone3} 2020-07-23T12:40:36.039 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-41 zones={test-zone2, test-zone3} 2020-07-23T12:40:36.052 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:36.098 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:36.162 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-41 zones={test-zone2, test-zone3} 2020-07-23T12:40:36.163 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-42 zones={test-zone2, test-zone1} 2020-07-23T12:40:36.174 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:36.222 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:36.294 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-42 zones={test-zone2, test-zone1} 2020-07-23T12:40:36.295 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-42 zones={test-zone3, test-zone1} 2020-07-23T12:40:36.309 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:36.358 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:36.429 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-42 zones={test-zone3, test-zone1} 2020-07-23T12:40:36.441 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-42 zones={test-zone1, test-zone2} 2020-07-23T12:40:36.460 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:36.510 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:36.572 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-42 zones={test-zone1, test-zone2} 2020-07-23T12:40:36.573 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-42 zones={test-zone3, test-zone2} 2020-07-23T12:40:36.584 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:36.638 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:36.709 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-42 zones={test-zone3, test-zone2} 2020-07-23T12:40:36.709 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-42 zones={test-zone1, test-zone3} 2020-07-23T12:40:36.722 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:36.770 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:36.833 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-42 zones={test-zone1, test-zone3} 2020-07-23T12:40:36.833 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-42 zones={test-zone2, test-zone3} 2020-07-23T12:40:36.844 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:36.890 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:36.953 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-42 zones={test-zone2, test-zone3} 2020-07-23T12:40:36.954 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-43 zones={test-zone2, test-zone1} 2020-07-23T12:40:36.965 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:37.014 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:37.079 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-43 zones={test-zone2, test-zone1} 2020-07-23T12:40:37.079 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-43 zones={test-zone3, test-zone1} 2020-07-23T12:40:37.088 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:37.134 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:37.198 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-43 zones={test-zone3, test-zone1} 2020-07-23T12:40:37.199 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-43 zones={test-zone1, test-zone2} 2020-07-23T12:40:37.210 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:37.258 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:37.321 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-43 zones={test-zone1, test-zone2} 2020-07-23T12:40:37.322 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-43 zones={test-zone3, test-zone2} 2020-07-23T12:40:37.332 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:37.382 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:37.445 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-43 zones={test-zone3, test-zone2} 2020-07-23T12:40:37.446 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-43 zones={test-zone1, test-zone3} 2020-07-23T12:40:37.467 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:37.514 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:37.579 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-43 zones={test-zone1, test-zone3} 2020-07-23T12:40:37.580 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-43 zones={test-zone2, test-zone3} 2020-07-23T12:40:37.593 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:37.646 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:37.710 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-43 zones={test-zone2, test-zone3} 2020-07-23T12:40:37.711 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-44 zones={test-zone2, test-zone1} 2020-07-23T12:40:37.721 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:37.770 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:37.838 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-44 zones={test-zone2, test-zone1} 2020-07-23T12:40:37.839 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-44 zones={test-zone3, test-zone1} 2020-07-23T12:40:37.848 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:37.898 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:37.961 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-44 zones={test-zone3, test-zone1} 2020-07-23T12:40:37.962 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-44 zones={test-zone1, test-zone2} 2020-07-23T12:40:37.973 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:38.026 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:38.100 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-44 zones={test-zone1, test-zone2} 2020-07-23T12:40:38.100 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-44 zones={test-zone3, test-zone2} 2020-07-23T12:40:38.111 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:38.158 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:38.224 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-44 zones={test-zone3, test-zone2} 2020-07-23T12:40:38.224 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-44 zones={test-zone1, test-zone3} 2020-07-23T12:40:38.238 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:38.286 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:38.352 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-44 zones={test-zone1, test-zone3} 2020-07-23T12:40:38.353 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-44 zones={test-zone2, test-zone3} 2020-07-23T12:40:38.365 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:38.410 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:38.474 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-44 zones={test-zone2, test-zone3} 2020-07-23T12:40:38.474 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-45 zones={test-zone2, test-zone1} 2020-07-23T12:40:38.485 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:38.534 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:38.602 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-45 zones={test-zone2, test-zone1} 2020-07-23T12:40:38.602 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-45 zones={test-zone3, test-zone1} 2020-07-23T12:40:38.612 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:38.658 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:38.722 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-45 zones={test-zone3, test-zone1} 2020-07-23T12:40:38.723 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-45 zones={test-zone1, test-zone2} 2020-07-23T12:40:38.734 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:38.786 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:38.857 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-45 zones={test-zone1, test-zone2} 2020-07-23T12:40:38.858 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-45 zones={test-zone3, test-zone2} 2020-07-23T12:40:38.868 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:38.918 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:38.982 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-45 zones={test-zone3, test-zone2} 2020-07-23T12:40:38.982 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-45 zones={test-zone1, test-zone3} 2020-07-23T12:40:38.996 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:39.042 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:39.109 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-45 zones={test-zone1, test-zone3} 2020-07-23T12:40:39.110 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-45 zones={test-zone2, test-zone3} 2020-07-23T12:40:39.120 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:40:39.170 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:40:39.234 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-45 zones={test-zone2, test-zone3} 2020-07-23T12:40:39.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '03aec9f8-1b80-4c7c-ae43-862168b0b5e6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:39.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '03aec9f8-1b80-4c7c-ae43-862168b0b5e6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:39.236 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 03aec9f8-1b80-4c7c-ae43-862168b0b5e6 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:39.275 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:39.275 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:39.539 INFO:teuthology.orchestra.run.smithi137.stdout:[{"id":"1_1595507375.318930_70.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-07-23T12:29:35.318930578Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2OfTQlQsQJ0GelLqSHyOAD_","ver":1},"status":{"status":"write"}}},{"id":"1_1595507375.329399_72.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-07-23T12:29:35.329399636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2OfTQlQsQJ0GelLqSHyOAD_","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507497.057065_74.1","section":"bucket.instance","name":"pdivcf-1:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.1","timestamp":"2020-07-23T12:31:37.057065382Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hxRQqFmO0OnO_LR85o5kUm5","ver":1},"status":{"status":"write"}}},{"id":"1_1595507497.061620_81.1","section":"bucket.instance","name":"pdivcf-1:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.1","timestamp":"2020-07-23T12:31:37.061620180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hxRQqFmO0OnO_LR85o5kUm5","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507497.063316_82.1","section":"bucket","name":"pdivcf-1","timestamp":"2020-07-23T12:31:37.063316155Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UlXS-dQKoitKY8vDjVB-gm8","ver":1},"status":{"status":"write"}}},{"id":"1_1595507497.067270_83.1","section":"bucket","name":"pdivcf-1","timestamp":"2020-07-23T12:31:37.067270716Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UlXS-dQKoitKY8vDjVB-gm8","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507497.098617_84.1","section":"bucket.instance","name":"pdivcf-2:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.2","timestamp":"2020-07-23T12:31:37.098617552Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AdQe82Xjsc-77q3kWXvfwP2","ver":1},"status":{"status":"write"}}},{"id":"1_1595507497.103291_85.1","section":"bucket.instance","name":"pdivcf-2:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.2","timestamp":"2020-07-23T12:31:37.103291974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AdQe82Xjsc-77q3kWXvfwP2","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507497.105451_86.1","section":"bucket","name":"pdivcf-2","timestamp":"2020-07-23T12:31:37.105451457Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FCKQTH1hiZp42khe9N5KHkG","ver":1},"status":{"status":"write"}}},{"id":"1_1595507497.110118_87.1","section":"bucket","name":"pdivcf-2","timestamp":"2020-07-23T12:31:37.110118928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FCKQTH1hiZp42khe9N5KHkG","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507500.591919_88.1","section":"bucket.instance","name":"pdivcf-3:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.3","timestamp":"2020-07-23T12:31:40.591919333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gQa_g97kTC2nL8PUml-gQsM","ver":1},"status":{"status":"write"}}},{"id":"1_1595507500.598942_89.1","section":"bucket.instance","name":"pdivcf-3:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.3","timestamp":"2020-07-23T12:31:40.598942621Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gQa_g97kTC2nL8PUml-gQsM","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507500.600878_90.1","section":"bucket","name":"pdivcf-3","timestamp":"2020-07-23T12:31:40.600878837Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5mXKhpLmaCwv2r1y08xfKZa","ver":1},"status":{"status":"write"}}},{"id":"1_1595507500.607977_91.1","section":"bucket","name":"pdivcf-3","timestamp":"2020-07-23T12:31:40.607977491Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5mXKhpLmaCwv2r1y08xfKZa","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.307745_92.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.4","timestamp":"2020-07-23T12:31:48.307745235Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yiZDGQu6X2F_Sk1M_XNX-aS","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.312319_95.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.4","timestamp":"2020-07-23T12:31:48.312319651Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yiZDGQu6X2F_Sk1M_XNX-aS","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.314524_96.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:48.314524642Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-Zgx0qRtoCp4uqoB0K4EqGk","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.318787_97.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:48.318787051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-Zgx0qRtoCp4uqoB0K4EqGk","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.342772_98.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.5","timestamp":"2020-07-23T12:31:48.342772112Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B4RZ8XZp_Vg_9H0AveP9SKb","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.347206_99.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.5","timestamp":"2020-07-23T12:31:48.347206077Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B4RZ8XZp_Vg_9H0AveP9SKb","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.349032_100.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:48.349032760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QSYVyw8rlDouP071b5aAg5c","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.352670_101.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:48.352670985Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QSYVyw8rlDouP071b5aAg5c","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.397942_102.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.6","timestamp":"2020-07-23T12:31:48.397942141Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Mxbry6a9nS2zmN7QABtJkqL","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.401908_103.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.6","timestamp":"2020-07-23T12:31:48.401908451Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Mxbry6a9nS2zmN7QABtJkqL","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.403730_104.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:48.403730015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nZJBxr1kdQBONurJVLbNJKj","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.407314_105.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:48.407314121Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nZJBxr1kdQBONurJVLbNJKj","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507515.909191_106.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.7","timestamp":"2020-07-23T12:31:55.909191875Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zoMNfl3WP6x69sG1A_a6fcE","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.913770_109.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.7","timestamp":"2020-07-23T12:31:55.913770150Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zoMNfl3WP6x69sG1A_a6fcE","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507515.915751_110.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:55.915751293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hU0LuGT_w40iAYldvPFrm0U","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.919252_111.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:55.919252707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hU0LuGT_w40iAYldvPFrm0U","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507515.950100_112.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.8","timestamp":"2020-07-23T12:31:55.950100507Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vltGzevaBLh8OZIhYyfm2CQ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.955276_113.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.8","timestamp":"2020-07-23T12:31:55.955276463Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vltGzevaBLh8OZIhYyfm2CQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507515.957617_114.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:55.957617299Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5r8WP0KmkR2EPhut7rsITD0","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.961660_115.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:55.961660024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5r8WP0KmkR2EPhut7rsITD0","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507515.989920_116.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.9","timestamp":"2020-07-23T12:31:55.989920273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zba4HrvvYXLNZLUZEEXyOjc","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.994394_117.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.9","timestamp":"2020-07-23T12:31:55.994394006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zba4HrvvYXLNZLUZEEXyOjc","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507515.996737_118.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:55.996737991Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XW9KXwz8j8Q2kDtlnNWtenR","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.002204_119.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.002204875Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XW9KXwz8j8Q2kDtlnNWtenR","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.038441_120.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.10","timestamp":"2020-07-23T12:31:56.038441936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yDod0geU58KV83pV2GT216-","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.042090_121.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.10","timestamp":"2020-07-23T12:31:56.042090624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yDod0geU58KV83pV2GT216-","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.044442_122.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:56.044442178Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SGxfEM8xG5yvSaheRQV2Ovd","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.047620_123.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:56.047620642Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SGxfEM8xG5yvSaheRQV2Ovd","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.098405_124.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.11","timestamp":"2020-07-23T12:31:56.098405415Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LUgRUmwvdm7gjw6CNXowxWA","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.102118_125.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.11","timestamp":"2020-07-23T12:31:56.102118377Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LUgRUmwvdm7gjw6CNXowxWA","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.103855_126.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:56.103855386Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_toR3dIX1U5fHjFxw5OD4lwp","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.106886_127.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:56.106886504Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_toR3dIX1U5fHjFxw5OD4lwp","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.198726_128.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.12","timestamp":"2020-07-23T12:31:56.198726163Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GxVDpKjDuPlGcusHQ2sPoDq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.202432_129.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.12","timestamp":"2020-07-23T12:31:56.202432137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GxVDpKjDuPlGcusHQ2sPoDq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.204287_130.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.204287823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wp9w1kaF3q9LFeeVvxwV4ZY","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.208680_131.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.208680506Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wp9w1kaF3q9LFeeVvxwV4ZY","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.310234_132.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.13","timestamp":"2020-07-23T12:31:56.310234859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pPnegWR9d5Ygh1JBGHNMLEp","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.314509_133.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.13","timestamp":"2020-07-23T12:31:56.314509529Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pPnegWR9d5Ygh1JBGHNMLEp","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.316454_134.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:56.316454484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a0pbGO-wJ9KDNLVLy0l6mVw","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.321088_135.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:56.321088724Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a0pbGO-wJ9KDNLVLy0l6mVw","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.376988_136.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.14","timestamp":"2020-07-23T12:31:56.376988037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5V1Wp4Ls0rjjCcGdYUoXxNK","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.381533_137.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.14","timestamp":"2020-07-23T12:31:56.381533072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5V1Wp4Ls0rjjCcGdYUoXxNK","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.383610_138.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:56.383610350Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qC-7doEAfyxvcolW3WYIsNi","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.387474_139.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:56.387474732Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qC-7doEAfyxvcolW3WYIsNi","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.482937_140.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.15","timestamp":"2020-07-23T12:31:56.482937943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6bW-esK6eOiMXZmLFiEt8dd","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.487703_141.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.15","timestamp":"2020-07-23T12:31:56.487703511Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6bW-esK6eOiMXZmLFiEt8dd","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.489723_142.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.489723477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RXg1UMJCNsOx1Xd-Urfc0kP","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.493752_143.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.493752921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RXg1UMJCNsOx1Xd-Urfc0kP","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507519.200871_144.1","section":"bucket.instance","name":"pdivcf-7:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.16","timestamp":"2020-07-23T12:31:59.200871054Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5kZ1JmFLTWYDEE5vSb3YN9q","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.205867_147.1","section":"bucket.instance","name":"pdivcf-7:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.16","timestamp":"2020-07-23T12:31:59.205867247Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5kZ1JmFLTWYDEE5vSb3YN9q","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.208273_148.1","section":"bucket","name":"pdivcf-7","timestamp":"2020-07-23T12:31:59.208273943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.213060_149.1","section":"bucket","name":"pdivcf-7","timestamp":"2020-07-23T12:31:59.213060668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.236100_150.1","section":"bucket.instance","name":"pdivcf-8:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.17","timestamp":"2020-07-23T12:31:59.236100590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tCGnu2GnZ4klKs-yTkLNnbJ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.242796_151.1","section":"bucket.instance","name":"pdivcf-8:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.17","timestamp":"2020-07-23T12:31:59.242796192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tCGnu2GnZ4klKs-yTkLNnbJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.246769_152.1","section":"bucket","name":"pdivcf-8","timestamp":"2020-07-23T12:31:59.246769828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.253139_153.1","section":"bucket","name":"pdivcf-8","timestamp":"2020-07-23T12:31:59.253139879Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.306952_154.1","section":"bucket.instance","name":"pdivcf-9:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.18","timestamp":"2020-07-23T12:31:59.306952465Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kbWQDeLkSgvmeNGJdVE78Au","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.312907_155.1","section":"bucket.instance","name":"pdivcf-9:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.18","timestamp":"2020-07-23T12:31:59.312907044Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kbWQDeLkSgvmeNGJdVE78Au","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.315224_156.1","section":"bucket","name":"pdivcf-9","timestamp":"2020-07-23T12:31:59.315224409Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.320666_157.1","section":"bucket","name":"pdivcf-9","timestamp":"2020-07-23T12:31:59.320666771Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507528.014327_158.1","section":"bucket","name":"pdivcf-7","timestamp":"2020-07-23T12:32:08.014327176Z","data":{"read_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":1},"write_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":2},"status":{"status":"remove"}}},{"id":"1_1595507528.020271_161.1","section":"bucket","name":"pdivcf-7","timestamp":"2020-07-23T12:32:08.020271414Z","data":{"read_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":1},"write_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507528.077441_162.1","section":"bucket","name":"pdivcf-8","timestamp":"2020-07-23T12:32:08.077441314Z","data":{"read_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":1},"write_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":2},"status":{"status":"remove"}}},{"id":"1_1595507528.083398_163.1","section":"bucket","name":"pdivcf-8","timestamp":"2020-07-23T12:32:08.083398771Z","data":{"read_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":1},"write_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507528.169614_164.1","section":"bucket","name":"pdivcf-9","timestamp":"2020-07-23T12:32:08.169614550Z","data":{"read_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":1},"write_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":2},"status":{"status":"remove"}}},{"id":"1_1595507528.176421_165.1","section":"bucket","name":"pdivcf-9","timestamp":"2020-07-23T12:32:08.176421339Z","data":{"read_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":1},"write_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507537.173491_46.1","section":"bucket.instance","name":"pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19","timestamp":"2020-07-23T12:32:17.173491511Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N7MWXq20eARG4bplabEWJWI","ver":1},"status":{"status":"write"}}},{"id":"1_1595507537.178224_55.1","section":"bucket.instance","name":"pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19","timestamp":"2020-07-23T12:32:17.178224061Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N7MWXq20eARG4bplabEWJWI","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507537.180408_56.1","section":"bucket","name":"pdivcf-10","timestamp":"2020-07-23T12:32:17.180408142Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eWs9kH85eL7ExiBJ3M7eDWt","ver":1},"status":{"status":"write"}}},{"id":"1_1595507537.185066_57.1","section":"bucket","name":"pdivcf-10","timestamp":"2020-07-23T12:32:17.185066160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eWs9kH85eL7ExiBJ3M7eDWt","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507538.116775_58.1","section":"bucket.instance","name":"pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20","timestamp":"2020-07-23T12:32:18.116775021Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0d0yAZmihVZhaW7NV6g_3kX","ver":1},"status":{"status":"write"}}},{"id":"1_1595507538.121315_59.1","section":"bucket.instance","name":"pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20","timestamp":"2020-07-23T12:32:18.121315724Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0d0yAZmihVZhaW7NV6g_3kX","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507538.123177_60.1","section":"bucket","name":"pdivcf-11","timestamp":"2020-07-23T12:32:18.123177285Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E1SDCqpJIo2DB_cVYeRt5Fj","ver":1},"status":{"status":"write"}}},{"id":"1_1595507538.130212_61.1","section":"bucket","name":"pdivcf-11","timestamp":"2020-07-23T12:32:18.130212887Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E1SDCqpJIo2DB_cVYeRt5Fj","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507538.842232_62.1","section":"bucket.instance","name":"pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21","timestamp":"2020-07-23T12:32:18.842232990Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DSr1_h9uAWM0Aupnq07MG-T","ver":1},"status":{"status":"write"}}},{"id":"1_1595507538.846555_63.1","section":"bucket.instance","name":"pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21","timestamp":"2020-07-23T12:32:18.846555026Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DSr1_h9uAWM0Aupnq07MG-T","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507538.848627_64.1","section":"bucket","name":"pdivcf-12","timestamp":"2020-07-23T12:32:18.848627622Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Idw7iMdTOFPnkuR0PvQ7oMk","ver":1},"status":{"status":"write"}}},{"id":"1_1595507538.853263_65.1","section":"bucket","name":"pdivcf-12","timestamp":"2020-07-23T12:32:18.853263990Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Idw7iMdTOFPnkuR0PvQ7oMk","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.772196_66.1","section":"bucket.instance","name":"pdivcf-13:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22","timestamp":"2020-07-23T12:32:37.772196573Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Na-elmOmjZsaFeM-ChUYR-E","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.776481_73.1","section":"bucket.instance","name":"pdivcf-13:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22","timestamp":"2020-07-23T12:32:37.776481061Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Na-elmOmjZsaFeM-ChUYR-E","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.778958_74.1","section":"bucket","name":"pdivcf-13","timestamp":"2020-07-23T12:32:37.778958954Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WAdUfnwKp-iQ-htLlTSCyaD","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.787127_75.1","section":"bucket","name":"pdivcf-13","timestamp":"2020-07-23T12:32:37.787127121Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WAdUfnwKp-iQ-htLlTSCyaD","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.815884_76.1","section":"bucket.instance","name":"pdivcf-14:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23","timestamp":"2020-07-23T12:32:37.815884493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BerFUu6TepMUW1OCfDzLt1L","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.820651_77.1","section":"bucket.instance","name":"pdivcf-14:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23","timestamp":"2020-07-23T12:32:37.820651075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BerFUu6TepMUW1OCfDzLt1L","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.823023_78.1","section":"bucket","name":"pdivcf-14","timestamp":"2020-07-23T12:32:37.823023440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBn5niBkGEfAMwLcYB5EkPd","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.828030_79.1","section":"bucket","name":"pdivcf-14","timestamp":"2020-07-23T12:32:37.828030042Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBn5niBkGEfAMwLcYB5EkPd","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.877028_80.1","section":"bucket.instance","name":"pdivcf-15:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24","timestamp":"2020-07-23T12:32:37.877028266Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ph9awnaTrqKViZtfefg5dn8","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.881682_81.1","section":"bucket.instance","name":"pdivcf-15:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24","timestamp":"2020-07-23T12:32:37.881682539Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ph9awnaTrqKViZtfefg5dn8","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.884047_82.1","section":"bucket","name":"pdivcf-15","timestamp":"2020-07-23T12:32:37.884047505Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HaDbmpKCs0MvAJEp019L9md","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.889012_83.1","section":"bucket","name":"pdivcf-15","timestamp":"2020-07-23T12:32:37.889012624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HaDbmpKCs0MvAJEp019L9md","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.476318_84.1","section":"bucket.instance","name":"pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25","timestamp":"2020-07-23T12:33:20.476318852Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.483791_98.1","section":"bucket.instance","name":"pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25","timestamp":"2020-07-23T12:33:20.483791941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.485654_99.1","section":"bucket","name":"pdivcf-16","timestamp":"2020-07-23T12:33:20.485654983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nNWx1-v33Cy8kx20mfEqCWM","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.490011_100.1","section":"bucket","name":"pdivcf-16","timestamp":"2020-07-23T12:33:20.490011274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nNWx1-v33Cy8kx20mfEqCWM","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.519518_101.1","section":"bucket.instance","name":"pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26","timestamp":"2020-07-23T12:33:20.519518015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.523662_102.1","section":"bucket.instance","name":"pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26","timestamp":"2020-07-23T12:33:20.523662515Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.526745_103.1","section":"bucket","name":"pdivcf-17","timestamp":"2020-07-23T12:33:20.526745182Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D9-gYM6k71SVeZ0yqDAlNyx","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.530907_104.1","section":"bucket","name":"pdivcf-17","timestamp":"2020-07-23T12:33:20.530907013Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D9-gYM6k71SVeZ0yqDAlNyx","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.570146_105.1","section":"bucket.instance","name":"pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27","timestamp":"2020-07-23T12:33:20.570146983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.575256_106.1","section":"bucket.instance","name":"pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27","timestamp":"2020-07-23T12:33:20.575256769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.577552_107.1","section":"bucket","name":"pdivcf-18","timestamp":"2020-07-23T12:33:20.577552876Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qfMtvuZVQROJ2UnUnTmsXWF","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.582467_108.1","section":"bucket","name":"pdivcf-18","timestamp":"2020-07-23T12:33:20.582467189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qfMtvuZVQROJ2UnUnTmsXWF","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.607611_109.1","section":"bucket.instance","name":"pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25","timestamp":"2020-07-23T12:33:20.607611890Z","data":{"read_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":1},"write_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":2},"status":{"status":"write"}}},{"id":"1_1595507600.614045_110.1","section":"bucket.instance","name":"pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25","timestamp":"2020-07-23T12:33:20.614045770Z","data":{"read_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":1},"write_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507600.619455_111.1","section":"bucket.instance","name":"pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26","timestamp":"2020-07-23T12:33:20.619455446Z","data":{"read_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":1},"write_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":2},"status":{"status":"write"}}},{"id":"1_1595507600.625710_112.1","section":"bucket.instance","name":"pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26","timestamp":"2020-07-23T12:33:20.625710444Z","data":{"read_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":1},"write_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507600.638571_113.1","section":"bucket.instance","name":"pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27","timestamp":"2020-07-23T12:33:20.638571023Z","data":{"read_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":1},"write_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":2},"status":{"status":"write"}}},{"id":"1_1595507600.647133_114.1","section":"bucket.instance","name":"pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27","timestamp":"2020-07-23T12:33:20.647133208Z","data":{"read_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":1},"write_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507704.629540_179.1","section":"bucket.instance","name":"pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28","timestamp":"2020-07-23T12:35:04.629540333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":1},"status":{"status":"write"}}},{"id":"1_1595507704.634646_185.1","section":"bucket.instance","name":"pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28","timestamp":"2020-07-23T12:35:04.634646116Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507704.636917_186.1","section":"bucket","name":"pdivcf-19","timestamp":"2020-07-23T12:35:04.636917194Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5lux1Kcalu1aSjCSsZbbPBc","ver":1},"status":{"status":"write"}}},{"id":"1_1595507704.641982_187.1","section":"bucket","name":"pdivcf-19","timestamp":"2020-07-23T12:35:04.641982741Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5lux1Kcalu1aSjCSsZbbPBc","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507704.648928_188.1","section":"bucket.instance","name":"pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28","timestamp":"2020-07-23T12:35:04.648928543Z","data":{"read_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":1},"write_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":2},"status":{"status":"write"}}},{"id":"1_1595507704.656778_189.1","section":"bucket.instance","name":"pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28","timestamp":"2020-07-23T12:35:04.656778355Z","data":{"read_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":1},"write_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507770.568084_190.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:10.568084858Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":1},"status":{"status":"write"}}},{"id":"1_1595507770.572580_207.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:10.572580901Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507770.574547_208.1","section":"bucket","name":"pdivcf-20","timestamp":"2020-07-23T12:36:10.574547102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1GieWDzPbyhzYx6dzAbvM8Y","ver":1},"status":{"status":"write"}}},{"id":"1_1595507770.579148_209.1","section":"bucket","name":"pdivcf-20","timestamp":"2020-07-23T12:36:10.579148822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1GieWDzPbyhzYx6dzAbvM8Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507790.368503_210.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:30.368503148Z","data":{"read_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":1},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":2},"status":{"status":"write"}}},{"id":"1_1595507790.379387_217.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:30.379387206Z","data":{"read_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":1},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507811.952039_218.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:51.952039133Z","data":{"read_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":2},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":3},"status":{"status":"write"}}},{"id":"1_1595507811.963376_225.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:51.963376890Z","data":{"read_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":2},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":3},"status":{"status":"complete"}}},{"id":"1_1595507833.977530_226.1","section":"bucket.instance","name":"pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30","timestamp":"2020-07-23T12:37:13.977530333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":1},"status":{"status":"write"}}},{"id":"1_1595507833.982409_233.1","section":"bucket.instance","name":"pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30","timestamp":"2020-07-23T12:37:13.982409662Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507833.984578_234.1","section":"bucket","name":"pdivcf-21","timestamp":"2020-07-23T12:37:13.984578572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxoN5QnXB8PzBQb6usiTCES","ver":1},"status":{"status":"write"}}},{"id":"1_1595507833.990265_235.1","section":"bucket","name":"pdivcf-21","timestamp":"2020-07-23T12:37:13.990265263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxoN5QnXB8PzBQb6usiTCES","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.016369_236.1","section":"bucket.instance","name":"pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31","timestamp":"2020-07-23T12:37:14.016369578Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507834.021064_237.1","section":"bucket.instance","name":"pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31","timestamp":"2020-07-23T12:37:14.021064652Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.023440_238.1","section":"bucket","name":"pdivcf-22","timestamp":"2020-07-23T12:37:14.023440018Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mOYhoW_mjO7mRP8PVQ6ZRH7","ver":1},"status":{"status":"write"}}},{"id":"1_1595507834.027916_239.1","section":"bucket","name":"pdivcf-22","timestamp":"2020-07-23T12:37:14.027916899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mOYhoW_mjO7mRP8PVQ6ZRH7","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.075022_240.1","section":"bucket.instance","name":"pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32","timestamp":"2020-07-23T12:37:14.075022229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":1},"status":{"status":"write"}}},{"id":"1_1595507834.080077_241.1","section":"bucket.instance","name":"pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32","timestamp":"2020-07-23T12:37:14.080077001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.082206_242.1","section":"bucket","name":"pdivcf-23","timestamp":"2020-07-23T12:37:14.082206069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C1euffFueI9tYW5vS7vMsgn","ver":1},"status":{"status":"write"}}},{"id":"1_1595507834.087443_243.1","section":"bucket","name":"pdivcf-23","timestamp":"2020-07-23T12:37:14.087443944Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C1euffFueI9tYW5vS7vMsgn","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.108675_244.1","section":"bucket.instance","name":"pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30","timestamp":"2020-07-23T12:37:14.108675749Z","data":{"read_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":1},"write_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":2},"status":{"status":"write"}}},{"id":"1_1595507834.113504_245.1","section":"bucket.instance","name":"pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30","timestamp":"2020-07-23T12:37:14.113504302Z","data":{"read_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":1},"write_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507834.118829_246.1","section":"bucket.instance","name":"pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31","timestamp":"2020-07-23T12:37:14.118829894Z","data":{"read_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":1},"write_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":2},"status":{"status":"write"}}},{"id":"1_1595507834.124671_247.1","section":"bucket.instance","name":"pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31","timestamp":"2020-07-23T12:37:14.124671872Z","data":{"read_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":1},"write_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507834.135668_248.1","section":"bucket.instance","name":"pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32","timestamp":"2020-07-23T12:37:14.135668915Z","data":{"read_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":1},"write_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":2},"status":{"status":"write"}}},{"id":"1_1595507834.142058_249.1","section":"bucket.instance","name":"pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32","timestamp":"2020-07-23T12:37:14.142058558Z","data":{"read_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":1},"write_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507862.301266_262.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.301266177Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.305752_275.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.305752811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.308557_276.1","section":"bucket","name":"pdivcf-24","timestamp":"2020-07-23T12:37:42.308557752Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9CmPcMhCq3DoG5OyzHLxsZp","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.313301_277.1","section":"bucket","name":"pdivcf-24","timestamp":"2020-07-23T12:37:42.313301958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9CmPcMhCq3DoG5OyzHLxsZp","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.341948_278.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.341948862Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.346312_279.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.346312178Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.348306_280.1","section":"bucket","name":"pdivcf-25","timestamp":"2020-07-23T12:37:42.348306812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_10rdqWmpSXM0gPEqJQLUt54","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.352897_281.1","section":"bucket","name":"pdivcf-25","timestamp":"2020-07-23T12:37:42.352897938Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_10rdqWmpSXM0gPEqJQLUt54","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.398608_282.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.398608049Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.402951_283.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.402951489Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.405025_284.1","section":"bucket","name":"pdivcf-26","timestamp":"2020-07-23T12:37:42.405025677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PtLBCckvoLRIiwjBffSRf4b","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.409024_285.1","section":"bucket","name":"pdivcf-26","timestamp":"2020-07-23T12:37:42.409024918Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PtLBCckvoLRIiwjBffSRf4b","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.436242_286.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.436242202Z","data":{"read_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":1},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":2},"status":{"status":"write"}}},{"id":"1_1595507862.443293_287.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.443293817Z","data":{"read_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":1},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507862.447803_288.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.447803976Z","data":{"read_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":2},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":3},"status":{"status":"write"}}},{"id":"1_1595507862.455267_289.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.455267808Z","data":{"read_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":2},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":3},"status":{"status":"complete"}}},{"id":"1_1595507862.461512_290.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.461512580Z","data":{"read_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":1},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":2},"status":{"status":"write"}}},{"id":"1_1595507862.469717_291.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.469717282Z","data":{"read_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":1},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507862.484236_292.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.484236274Z","data":{"read_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":2},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":3},"status":{"status":"write"}}},{"id":"1_1595507862.493648_293.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.493648043Z","data":{"read_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":2},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":3},"status":{"status":"complete"}}},{"id":"1_1595507862.509518_294.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.509518426Z","data":{"read_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":1},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":2},"status":{"status":"write"}}},{"id":"1_1595507862.519332_295.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.519332057Z","data":{"read_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":1},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507862.547588_296.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.547588345Z","data":{"read_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":2},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":3},"status":{"status":"write"}}},{"id":"1_1595507862.561233_297.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.561233909Z","data":{"read_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":2},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":3},"status":{"status":"complete"}}},{"id":"1_1595507882.721482_301.1","section":"bucket.instance","name":"pdivcf-27:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.36","timestamp":"2020-07-23T12:38:02.721482894Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.726106_317.1","section":"bucket.instance","name":"pdivcf-27:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.36","timestamp":"2020-07-23T12:38:02.726106004Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.728482_318.1","section":"bucket","name":"pdivcf-27","timestamp":"2020-07-23T12:38:02.728482658Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0hGeTHEMYPqYlW8vmo8ZsB4","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.733340_319.1","section":"bucket","name":"pdivcf-27","timestamp":"2020-07-23T12:38:02.733340176Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0hGeTHEMYPqYlW8vmo8ZsB4","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.764598_320.1","section":"bucket.instance","name":"pdivcf-28:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37","timestamp":"2020-07-23T12:38:02.764598242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.768937_321.1","section":"bucket.instance","name":"pdivcf-28:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37","timestamp":"2020-07-23T12:38:02.768937393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.771175_322.1","section":"bucket","name":"pdivcf-28","timestamp":"2020-07-23T12:38:02.771175166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7Rwx9HRnK8Xo_lM2eZ_spRn","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.776360_323.1","section":"bucket","name":"pdivcf-28","timestamp":"2020-07-23T12:38:02.776360587Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7Rwx9HRnK8Xo_lM2eZ_spRn","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.827591_324.1","section":"bucket.instance","name":"pdivcf-29:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38","timestamp":"2020-07-23T12:38:02.827591764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.832410_325.1","section":"bucket.instance","name":"pdivcf-29:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38","timestamp":"2020-07-23T12:38:02.832410818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.834625_326.1","section":"bucket","name":"pdivcf-29","timestamp":"2020-07-23T12:38:02.834625226Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FBpeaUcoiAUeypLntXcZ6mt","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.839185_327.1","section":"bucket","name":"pdivcf-29","timestamp":"2020-07-23T12:38:02.839185239Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FBpeaUcoiAUeypLntXcZ6mt","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.863209_328.1","section":"bucket.instance","name":"pdivcf-27:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.36","timestamp":"2020-07-23T12:38:02.863209772Z","data":{"read_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":1},"write_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":2},"status":{"status":"write"}}},{"id":"1_1595507882.870400_329.1","section":"bucket.instance","name":"pdivcf-27:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.36","timestamp":"2020-07-23T12:38:02.870400054Z","data":{"read_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":1},"write_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507882.925804_330.1","section":"bucket.instance","name":"pdivcf-28:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37","timestamp":"2020-07-23T12:38:02.925804207Z","data":{"read_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":1},"write_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":2},"status":{"status":"write"}}},{"id":"1_1595507882.936647_331.1","section":"bucket.instance","name":"pdivcf-28:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37","timestamp":"2020-07-23T12:38:02.936647924Z","data":{"read_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":1},"write_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507882.996474_332.1","section":"bucket.instance","name":"pdivcf-29:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38","timestamp":"2020-07-23T12:38:02.996474008Z","data":{"read_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":1},"write_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.007982_333.1","section":"bucket.instance","name":"pdivcf-29:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38","timestamp":"2020-07-23T12:38:03.007982412Z","data":{"read_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":1},"write_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507883.085277_334.1","section":"bucket.instance","name":"pdivcf-30:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.39","timestamp":"2020-07-23T12:38:03.085277631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.094971_335.1","section":"bucket.instance","name":"pdivcf-30:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.39","timestamp":"2020-07-23T12:38:03.094971334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.097157_336.1","section":"bucket","name":"pdivcf-30","timestamp":"2020-07-23T12:38:03.097157211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pkn26B7VC8HessJ3RMQLyfb","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.101146_337.1","section":"bucket","name":"pdivcf-30","timestamp":"2020-07-23T12:38:03.101146718Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pkn26B7VC8HessJ3RMQLyfb","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.123841_338.1","section":"bucket.instance","name":"pdivcf-31:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40","timestamp":"2020-07-23T12:38:03.123841176Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.128855_339.1","section":"bucket.instance","name":"pdivcf-31:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40","timestamp":"2020-07-23T12:38:03.128855661Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.131009_340.1","section":"bucket","name":"pdivcf-31","timestamp":"2020-07-23T12:38:03.131009569Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JpktLwZXqe4IoNIvo0sPRIm","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.135379_341.1","section":"bucket","name":"pdivcf-31","timestamp":"2020-07-23T12:38:03.135379648Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JpktLwZXqe4IoNIvo0sPRIm","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.179493_342.1","section":"bucket.instance","name":"pdivcf-32:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41","timestamp":"2020-07-23T12:38:03.179493154Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.183875_343.1","section":"bucket.instance","name":"pdivcf-32:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41","timestamp":"2020-07-23T12:38:03.183875771Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.186063_344.1","section":"bucket","name":"pdivcf-32","timestamp":"2020-07-23T12:38:03.186063685Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2yNrr1LVREKqlHu0zys3qhJ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.190110_345.1","section":"bucket","name":"pdivcf-32","timestamp":"2020-07-23T12:38:03.190110469Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2yNrr1LVREKqlHu0zys3qhJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.215750_346.1","section":"bucket.instance","name":"pdivcf-30:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.39","timestamp":"2020-07-23T12:38:03.215750124Z","data":{"read_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":1},"write_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.221277_347.1","section":"bucket.instance","name":"pdivcf-30:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.39","timestamp":"2020-07-23T12:38:03.221277172Z","data":{"read_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":1},"write_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507883.274318_348.1","section":"bucket.instance","name":"pdivcf-31:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40","timestamp":"2020-07-23T12:38:03.274318325Z","data":{"read_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":1},"write_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.286399_349.1","section":"bucket.instance","name":"pdivcf-31:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40","timestamp":"2020-07-23T12:38:03.286399254Z","data":{"read_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":1},"write_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507883.346515_350.1","section":"bucket.instance","name":"pdivcf-32:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41","timestamp":"2020-07-23T12:38:03.346515051Z","data":{"read_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":1},"write_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.356462_351.1","section":"bucket.instance","name":"pdivcf-32:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41","timestamp":"2020-07-23T12:38:03.356462902Z","data":{"read_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":1},"write_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507883.435617_352.1","section":"bucket.instance","name":"pdivcf-33:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.42","timestamp":"2020-07-23T12:38:03.435617749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.449308_353.1","section":"bucket.instance","name":"pdivcf-33:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.42","timestamp":"2020-07-23T12:38:03.449308210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.462520_354.1","section":"bucket","name":"pdivcf-33","timestamp":"2020-07-23T12:38:03.462520643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TJwU15-56F6OaBLbx9XF7mq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.466784_355.1","section":"bucket","name":"pdivcf-33","timestamp":"2020-07-23T12:38:03.466784917Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TJwU15-56F6OaBLbx9XF7mq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.564568_356.1","section":"bucket.instance","name":"pdivcf-34:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43","timestamp":"2020-07-23T12:38:03.564568818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.569408_357.1","section":"bucket.instance","name":"pdivcf-34:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43","timestamp":"2020-07-23T12:38:03.569408997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.571379_358.1","section":"bucket","name":"pdivcf-34","timestamp":"2020-07-23T12:38:03.571379967Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QaQnidDCu-eD3G2dQaeWRxh","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.575816_359.1","section":"bucket","name":"pdivcf-34","timestamp":"2020-07-23T12:38:03.575816752Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QaQnidDCu-eD3G2dQaeWRxh","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.617917_360.1","section":"bucket.instance","name":"pdivcf-35:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44","timestamp":"2020-07-23T12:38:03.617917681Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.622832_361.1","section":"bucket.instance","name":"pdivcf-35:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44","timestamp":"2020-07-23T12:38:03.622832163Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.625040_362.1","section":"bucket","name":"pdivcf-35","timestamp":"2020-07-23T12:38:03.625040094Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f8RF8Y341PAQo2j4DYcRQfj","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.630457_363.1","section":"bucket","name":"pdivcf-35","timestamp":"2020-07-23T12:38:03.630457468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f8RF8Y341PAQo2j4DYcRQfj","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.656588_364.1","section":"bucket.instance","name":"pdivcf-33:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.42","timestamp":"2020-07-23T12:38:03.656588365Z","data":{"read_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":1},"write_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.663207_365.1","section":"bucket.instance","name":"pdivcf-33:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.42","timestamp":"2020-07-23T12:38:03.663207212Z","data":{"read_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":1},"write_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507884.713429_366.1","section":"bucket.instance","name":"pdivcf-34:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43","timestamp":"2020-07-23T12:38:04.713429561Z","data":{"read_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":1},"write_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":2},"status":{"status":"write"}}},{"id":"1_1595507884.725262_367.1","section":"bucket.instance","name":"pdivcf-34:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43","timestamp":"2020-07-23T12:38:04.725262276Z","data":{"read_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":1},"write_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507885.864935_368.1","section":"bucket.instance","name":"pdivcf-35:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44","timestamp":"2020-07-23T12:38:05.864935216Z","data":{"read_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":1},"write_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":2},"status":{"status":"write"}}},{"id":"1_1595507885.876462_369.1","section":"bucket.instance","name":"pdivcf-35:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44","timestamp":"2020-07-23T12:38:05.876462797Z","data":{"read_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":1},"write_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507886.300008_370.1","section":"bucket.instance","name":"pdivcf-36:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45","timestamp":"2020-07-23T12:38:06.300008664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ps7FDMLj9P7FIJ3TQF3GFZk","ver":1},"status":{"status":"write"}}},{"id":"1_1595507886.304501_371.1","section":"bucket.instance","name":"pdivcf-36:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45","timestamp":"2020-07-23T12:38:06.304501769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ps7FDMLj9P7FIJ3TQF3GFZk","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507886.306748_372.1","section":"bucket","name":"pdivcf-36","timestamp":"2020-07-23T12:38:06.306748955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LrkNjaTY34zXjmNUserxO9w","ver":1},"status":{"status":"write"}}},{"id":"1_1595507886.311229_373.1","section":"bucket","name":"pdivcf-36","timestamp":"2020-07-23T12:38:06.311229165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LrkNjaTY34zXjmNUserxO9w","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507887.131065_374.1","section":"bucket.instance","name":"pdivcf-37:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46","timestamp":"2020-07-23T12:38:07.131065889Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Voz6xrKT44N52QQm_1jkHXq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507887.136159_375.1","section":"bucket.instance","name":"pdivcf-37:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46","timestamp":"2020-07-23T12:38:07.136159187Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Voz6xrKT44N52QQm_1jkHXq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507887.138199_376.1","section":"bucket","name":"pdivcf-37","timestamp":"2020-07-23T12:38:07.138199539Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GgAC1P5cFFfaEIRduZflSLw","ver":1},"status":{"status":"write"}}},{"id":"1_1595507887.142636_377.1","section":"bucket","name":"pdivcf-37","timestamp":"2020-07-23T12:38:07.142636002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GgAC1P5cFFfaEIRduZflSLw","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507887.514178_378.1","section":"bucket.instance","name":"pdivcf-38:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47","timestamp":"2020-07-23T12:38:07.514178897Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oY59xvqz8ppnd0uahVk4x_2","ver":1},"status":{"status":"write"}}},{"id":"1_1595507887.518533_379.1","section":"bucket.instance","name":"pdivcf-38:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47","timestamp":"2020-07-23T12:38:07.518533828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oY59xvqz8ppnd0uahVk4x_2","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507887.520752_380.1","section":"bucket","name":"pdivcf-38","timestamp":"2020-07-23T12:38:07.520752755Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N85MuH5ymUQx_bvR1jEV5oS","ver":1},"status":{"status":"write"}}},{"id":"1_1595507887.526314_381.1","section":"bucket","name":"pdivcf-38","timestamp":"2020-07-23T12:38:07.526314606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N85MuH5ymUQx_bvR1jEV5oS","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.085257_382.1","section":"bucket.instance","name":"pdivcf-39:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.48","timestamp":"2020-07-23T12:38:10.085257016Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_47jziFa-v3TzgKog19keFK3","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.090510_384.1","section":"bucket.instance","name":"pdivcf-39:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.48","timestamp":"2020-07-23T12:38:10.090510647Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_47jziFa-v3TzgKog19keFK3","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.092807_385.1","section":"bucket","name":"pdivcf-39","timestamp":"2020-07-23T12:38:10.092807212Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J6h6UEs-Kdv67K6lZ-DDqe_","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.098065_386.1","section":"bucket","name":"pdivcf-39","timestamp":"2020-07-23T12:38:10.098065606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J6h6UEs-Kdv67K6lZ-DDqe_","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.130438_387.1","section":"bucket.instance","name":"pdivcf-40:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.49","timestamp":"2020-07-23T12:38:10.130438151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jBkqIWgSeJygUbxw4jdVMVd","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.135039_388.1","section":"bucket.instance","name":"pdivcf-40:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.49","timestamp":"2020-07-23T12:38:10.135039327Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jBkqIWgSeJygUbxw4jdVMVd","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.137937_389.1","section":"bucket","name":"pdivcf-40","timestamp":"2020-07-23T12:38:10.137937297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Nsh2aQliOMNd_qa-DYEFjIr","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.146156_390.1","section":"bucket","name":"pdivcf-40","timestamp":"2020-07-23T12:38:10.146156264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Nsh2aQliOMNd_qa-DYEFjIr","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.204712_391.1","section":"bucket.instance","name":"pdivcf-41:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.50","timestamp":"2020-07-23T12:38:10.204712051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jjdhZX1cAX8QT7odbfHWP9O","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.209370_392.1","section":"bucket.instance","name":"pdivcf-41:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.50","timestamp":"2020-07-23T12:38:10.209370112Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jjdhZX1cAX8QT7odbfHWP9O","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.211763_393.1","section":"bucket","name":"pdivcf-41","timestamp":"2020-07-23T12:38:10.211763358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Voj1i0FYSZ0L7r0fMh4Tdjk","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.217061_394.1","section":"bucket","name":"pdivcf-41","timestamp":"2020-07-23T12:38:10.217061137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Voj1i0FYSZ0L7r0fMh4Tdjk","ver":1},"status":{"status":"complete"}}}] 2020-07-23T12:40:39.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '03aec9f8-1b80-4c7c-ae43-862168b0b5e6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:39.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '03aec9f8-1b80-4c7c-ae43-862168b0b5e6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:39.545 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 03aec9f8-1b80-4c7c-ae43-862168b0b5e6 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:39.586 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:39.586 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:39.847 INFO:teuthology.orchestra.run.smithi133.stdout:[{"id":"1_1595507375.318930_70.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-07-23T12:29:35.318930578Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2OfTQlQsQJ0GelLqSHyOAD_","ver":1},"status":{"status":"write"}}},{"id":"1_1595507375.329399_72.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-07-23T12:29:35.329399636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2OfTQlQsQJ0GelLqSHyOAD_","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507497.057065_74.1","section":"bucket.instance","name":"pdivcf-1:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.1","timestamp":"2020-07-23T12:31:37.057065382Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hxRQqFmO0OnO_LR85o5kUm5","ver":1},"status":{"status":"write"}}},{"id":"1_1595507497.061620_81.1","section":"bucket.instance","name":"pdivcf-1:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.1","timestamp":"2020-07-23T12:31:37.061620180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hxRQqFmO0OnO_LR85o5kUm5","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507497.063316_82.1","section":"bucket","name":"pdivcf-1","timestamp":"2020-07-23T12:31:37.063316155Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UlXS-dQKoitKY8vDjVB-gm8","ver":1},"status":{"status":"write"}}},{"id":"1_1595507497.067270_83.1","section":"bucket","name":"pdivcf-1","timestamp":"2020-07-23T12:31:37.067270716Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UlXS-dQKoitKY8vDjVB-gm8","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507497.098617_84.1","section":"bucket.instance","name":"pdivcf-2:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.2","timestamp":"2020-07-23T12:31:37.098617552Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AdQe82Xjsc-77q3kWXvfwP2","ver":1},"status":{"status":"write"}}},{"id":"1_1595507497.103291_85.1","section":"bucket.instance","name":"pdivcf-2:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.2","timestamp":"2020-07-23T12:31:37.103291974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AdQe82Xjsc-77q3kWXvfwP2","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507497.105451_86.1","section":"bucket","name":"pdivcf-2","timestamp":"2020-07-23T12:31:37.105451457Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FCKQTH1hiZp42khe9N5KHkG","ver":1},"status":{"status":"write"}}},{"id":"1_1595507497.110118_87.1","section":"bucket","name":"pdivcf-2","timestamp":"2020-07-23T12:31:37.110118928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FCKQTH1hiZp42khe9N5KHkG","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507500.591919_88.1","section":"bucket.instance","name":"pdivcf-3:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.3","timestamp":"2020-07-23T12:31:40.591919333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gQa_g97kTC2nL8PUml-gQsM","ver":1},"status":{"status":"write"}}},{"id":"1_1595507500.598942_89.1","section":"bucket.instance","name":"pdivcf-3:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.3","timestamp":"2020-07-23T12:31:40.598942621Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gQa_g97kTC2nL8PUml-gQsM","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507500.600878_90.1","section":"bucket","name":"pdivcf-3","timestamp":"2020-07-23T12:31:40.600878837Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5mXKhpLmaCwv2r1y08xfKZa","ver":1},"status":{"status":"write"}}},{"id":"1_1595507500.607977_91.1","section":"bucket","name":"pdivcf-3","timestamp":"2020-07-23T12:31:40.607977491Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5mXKhpLmaCwv2r1y08xfKZa","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.307745_92.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.4","timestamp":"2020-07-23T12:31:48.307745235Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yiZDGQu6X2F_Sk1M_XNX-aS","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.312319_95.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.4","timestamp":"2020-07-23T12:31:48.312319651Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yiZDGQu6X2F_Sk1M_XNX-aS","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.314524_96.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:48.314524642Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-Zgx0qRtoCp4uqoB0K4EqGk","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.318787_97.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:48.318787051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-Zgx0qRtoCp4uqoB0K4EqGk","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.342772_98.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.5","timestamp":"2020-07-23T12:31:48.342772112Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B4RZ8XZp_Vg_9H0AveP9SKb","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.347206_99.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.5","timestamp":"2020-07-23T12:31:48.347206077Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B4RZ8XZp_Vg_9H0AveP9SKb","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.349032_100.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:48.349032760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QSYVyw8rlDouP071b5aAg5c","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.352670_101.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:48.352670985Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QSYVyw8rlDouP071b5aAg5c","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.397942_102.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.6","timestamp":"2020-07-23T12:31:48.397942141Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Mxbry6a9nS2zmN7QABtJkqL","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.401908_103.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.6","timestamp":"2020-07-23T12:31:48.401908451Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Mxbry6a9nS2zmN7QABtJkqL","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.403730_104.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:48.403730015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nZJBxr1kdQBONurJVLbNJKj","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.407314_105.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:48.407314121Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nZJBxr1kdQBONurJVLbNJKj","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507515.909191_106.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.7","timestamp":"2020-07-23T12:31:55.909191875Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zoMNfl3WP6x69sG1A_a6fcE","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.913770_109.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.7","timestamp":"2020-07-23T12:31:55.913770150Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zoMNfl3WP6x69sG1A_a6fcE","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507515.915751_110.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:55.915751293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hU0LuGT_w40iAYldvPFrm0U","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.919252_111.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:55.919252707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hU0LuGT_w40iAYldvPFrm0U","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507515.950100_112.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.8","timestamp":"2020-07-23T12:31:55.950100507Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vltGzevaBLh8OZIhYyfm2CQ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.955276_113.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.8","timestamp":"2020-07-23T12:31:55.955276463Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vltGzevaBLh8OZIhYyfm2CQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507515.957617_114.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:55.957617299Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5r8WP0KmkR2EPhut7rsITD0","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.961660_115.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:55.961660024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5r8WP0KmkR2EPhut7rsITD0","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507515.989920_116.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.9","timestamp":"2020-07-23T12:31:55.989920273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zba4HrvvYXLNZLUZEEXyOjc","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.994394_117.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.9","timestamp":"2020-07-23T12:31:55.994394006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zba4HrvvYXLNZLUZEEXyOjc","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507515.996737_118.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:55.996737991Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XW9KXwz8j8Q2kDtlnNWtenR","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.002204_119.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.002204875Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XW9KXwz8j8Q2kDtlnNWtenR","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.038441_120.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.10","timestamp":"2020-07-23T12:31:56.038441936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yDod0geU58KV83pV2GT216-","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.042090_121.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.10","timestamp":"2020-07-23T12:31:56.042090624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yDod0geU58KV83pV2GT216-","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.044442_122.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:56.044442178Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SGxfEM8xG5yvSaheRQV2Ovd","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.047620_123.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:56.047620642Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SGxfEM8xG5yvSaheRQV2Ovd","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.098405_124.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.11","timestamp":"2020-07-23T12:31:56.098405415Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LUgRUmwvdm7gjw6CNXowxWA","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.102118_125.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.11","timestamp":"2020-07-23T12:31:56.102118377Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LUgRUmwvdm7gjw6CNXowxWA","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.103855_126.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:56.103855386Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_toR3dIX1U5fHjFxw5OD4lwp","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.106886_127.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:56.106886504Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_toR3dIX1U5fHjFxw5OD4lwp","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.198726_128.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.12","timestamp":"2020-07-23T12:31:56.198726163Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GxVDpKjDuPlGcusHQ2sPoDq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.202432_129.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.12","timestamp":"2020-07-23T12:31:56.202432137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GxVDpKjDuPlGcusHQ2sPoDq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.204287_130.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.204287823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wp9w1kaF3q9LFeeVvxwV4ZY","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.208680_131.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.208680506Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wp9w1kaF3q9LFeeVvxwV4ZY","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.310234_132.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.13","timestamp":"2020-07-23T12:31:56.310234859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pPnegWR9d5Ygh1JBGHNMLEp","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.314509_133.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.13","timestamp":"2020-07-23T12:31:56.314509529Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pPnegWR9d5Ygh1JBGHNMLEp","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.316454_134.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:56.316454484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a0pbGO-wJ9KDNLVLy0l6mVw","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.321088_135.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:56.321088724Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a0pbGO-wJ9KDNLVLy0l6mVw","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.376988_136.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.14","timestamp":"2020-07-23T12:31:56.376988037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5V1Wp4Ls0rjjCcGdYUoXxNK","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.381533_137.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.14","timestamp":"2020-07-23T12:31:56.381533072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5V1Wp4Ls0rjjCcGdYUoXxNK","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.383610_138.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:56.383610350Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qC-7doEAfyxvcolW3WYIsNi","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.387474_139.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:56.387474732Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qC-7doEAfyxvcolW3WYIsNi","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.482937_140.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.15","timestamp":"2020-07-23T12:31:56.482937943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6bW-esK6eOiMXZmLFiEt8dd","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.487703_141.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.15","timestamp":"2020-07-23T12:31:56.487703511Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6bW-esK6eOiMXZmLFiEt8dd","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.489723_142.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.489723477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RXg1UMJCNsOx1Xd-Urfc0kP","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.493752_143.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.493752921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RXg1UMJCNsOx1Xd-Urfc0kP","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507519.200871_144.1","section":"bucket.instance","name":"pdivcf-7:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.16","timestamp":"2020-07-23T12:31:59.200871054Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5kZ1JmFLTWYDEE5vSb3YN9q","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.205867_147.1","section":"bucket.instance","name":"pdivcf-7:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.16","timestamp":"2020-07-23T12:31:59.205867247Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5kZ1JmFLTWYDEE5vSb3YN9q","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.208273_148.1","section":"bucket","name":"pdivcf-7","timestamp":"2020-07-23T12:31:59.208273943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.213060_149.1","section":"bucket","name":"pdivcf-7","timestamp":"2020-07-23T12:31:59.213060668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.236100_150.1","section":"bucket.instance","name":"pdivcf-8:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.17","timestamp":"2020-07-23T12:31:59.236100590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tCGnu2GnZ4klKs-yTkLNnbJ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.242796_151.1","section":"bucket.instance","name":"pdivcf-8:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.17","timestamp":"2020-07-23T12:31:59.242796192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tCGnu2GnZ4klKs-yTkLNnbJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.246769_152.1","section":"bucket","name":"pdivcf-8","timestamp":"2020-07-23T12:31:59.246769828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.253139_153.1","section":"bucket","name":"pdivcf-8","timestamp":"2020-07-23T12:31:59.253139879Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.306952_154.1","section":"bucket.instance","name":"pdivcf-9:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.18","timestamp":"2020-07-23T12:31:59.306952465Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kbWQDeLkSgvmeNGJdVE78Au","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.312907_155.1","section":"bucket.instance","name":"pdivcf-9:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.18","timestamp":"2020-07-23T12:31:59.312907044Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kbWQDeLkSgvmeNGJdVE78Au","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.315224_156.1","section":"bucket","name":"pdivcf-9","timestamp":"2020-07-23T12:31:59.315224409Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.320666_157.1","section":"bucket","name":"pdivcf-9","timestamp":"2020-07-23T12:31:59.320666771Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507528.014327_158.1","section":"bucket","name":"pdivcf-7","timestamp":"2020-07-23T12:32:08.014327176Z","data":{"read_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":1},"write_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":2},"status":{"status":"remove"}}},{"id":"1_1595507528.020271_161.1","section":"bucket","name":"pdivcf-7","timestamp":"2020-07-23T12:32:08.020271414Z","data":{"read_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":1},"write_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507528.077441_162.1","section":"bucket","name":"pdivcf-8","timestamp":"2020-07-23T12:32:08.077441314Z","data":{"read_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":1},"write_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":2},"status":{"status":"remove"}}},{"id":"1_1595507528.083398_163.1","section":"bucket","name":"pdivcf-8","timestamp":"2020-07-23T12:32:08.083398771Z","data":{"read_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":1},"write_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507528.169614_164.1","section":"bucket","name":"pdivcf-9","timestamp":"2020-07-23T12:32:08.169614550Z","data":{"read_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":1},"write_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":2},"status":{"status":"remove"}}},{"id":"1_1595507528.176421_165.1","section":"bucket","name":"pdivcf-9","timestamp":"2020-07-23T12:32:08.176421339Z","data":{"read_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":1},"write_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507537.173491_46.1","section":"bucket.instance","name":"pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19","timestamp":"2020-07-23T12:32:17.173491511Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N7MWXq20eARG4bplabEWJWI","ver":1},"status":{"status":"write"}}},{"id":"1_1595507537.178224_55.1","section":"bucket.instance","name":"pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19","timestamp":"2020-07-23T12:32:17.178224061Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N7MWXq20eARG4bplabEWJWI","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507537.180408_56.1","section":"bucket","name":"pdivcf-10","timestamp":"2020-07-23T12:32:17.180408142Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eWs9kH85eL7ExiBJ3M7eDWt","ver":1},"status":{"status":"write"}}},{"id":"1_1595507537.185066_57.1","section":"bucket","name":"pdivcf-10","timestamp":"2020-07-23T12:32:17.185066160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eWs9kH85eL7ExiBJ3M7eDWt","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507538.116775_58.1","section":"bucket.instance","name":"pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20","timestamp":"2020-07-23T12:32:18.116775021Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0d0yAZmihVZhaW7NV6g_3kX","ver":1},"status":{"status":"write"}}},{"id":"1_1595507538.121315_59.1","section":"bucket.instance","name":"pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20","timestamp":"2020-07-23T12:32:18.121315724Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0d0yAZmihVZhaW7NV6g_3kX","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507538.123177_60.1","section":"bucket","name":"pdivcf-11","timestamp":"2020-07-23T12:32:18.123177285Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E1SDCqpJIo2DB_cVYeRt5Fj","ver":1},"status":{"status":"write"}}},{"id":"1_1595507538.130212_61.1","section":"bucket","name":"pdivcf-11","timestamp":"2020-07-23T12:32:18.130212887Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E1SDCqpJIo2DB_cVYeRt5Fj","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507538.842232_62.1","section":"bucket.instance","name":"pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21","timestamp":"2020-07-23T12:32:18.842232990Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DSr1_h9uAWM0Aupnq07MG-T","ver":1},"status":{"status":"write"}}},{"id":"1_1595507538.846555_63.1","section":"bucket.instance","name":"pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21","timestamp":"2020-07-23T12:32:18.846555026Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DSr1_h9uAWM0Aupnq07MG-T","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507538.848627_64.1","section":"bucket","name":"pdivcf-12","timestamp":"2020-07-23T12:32:18.848627622Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Idw7iMdTOFPnkuR0PvQ7oMk","ver":1},"status":{"status":"write"}}},{"id":"1_1595507538.853263_65.1","section":"bucket","name":"pdivcf-12","timestamp":"2020-07-23T12:32:18.853263990Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Idw7iMdTOFPnkuR0PvQ7oMk","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.772196_66.1","section":"bucket.instance","name":"pdivcf-13:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22","timestamp":"2020-07-23T12:32:37.772196573Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Na-elmOmjZsaFeM-ChUYR-E","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.776481_73.1","section":"bucket.instance","name":"pdivcf-13:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22","timestamp":"2020-07-23T12:32:37.776481061Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Na-elmOmjZsaFeM-ChUYR-E","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.778958_74.1","section":"bucket","name":"pdivcf-13","timestamp":"2020-07-23T12:32:37.778958954Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WAdUfnwKp-iQ-htLlTSCyaD","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.787127_75.1","section":"bucket","name":"pdivcf-13","timestamp":"2020-07-23T12:32:37.787127121Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WAdUfnwKp-iQ-htLlTSCyaD","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.815884_76.1","section":"bucket.instance","name":"pdivcf-14:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23","timestamp":"2020-07-23T12:32:37.815884493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BerFUu6TepMUW1OCfDzLt1L","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.820651_77.1","section":"bucket.instance","name":"pdivcf-14:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23","timestamp":"2020-07-23T12:32:37.820651075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BerFUu6TepMUW1OCfDzLt1L","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.823023_78.1","section":"bucket","name":"pdivcf-14","timestamp":"2020-07-23T12:32:37.823023440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBn5niBkGEfAMwLcYB5EkPd","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.828030_79.1","section":"bucket","name":"pdivcf-14","timestamp":"2020-07-23T12:32:37.828030042Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBn5niBkGEfAMwLcYB5EkPd","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.877028_80.1","section":"bucket.instance","name":"pdivcf-15:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24","timestamp":"2020-07-23T12:32:37.877028266Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ph9awnaTrqKViZtfefg5dn8","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.881682_81.1","section":"bucket.instance","name":"pdivcf-15:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24","timestamp":"2020-07-23T12:32:37.881682539Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ph9awnaTrqKViZtfefg5dn8","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.884047_82.1","section":"bucket","name":"pdivcf-15","timestamp":"2020-07-23T12:32:37.884047505Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HaDbmpKCs0MvAJEp019L9md","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.889012_83.1","section":"bucket","name":"pdivcf-15","timestamp":"2020-07-23T12:32:37.889012624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HaDbmpKCs0MvAJEp019L9md","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.476318_84.1","section":"bucket.instance","name":"pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25","timestamp":"2020-07-23T12:33:20.476318852Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.483791_98.1","section":"bucket.instance","name":"pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25","timestamp":"2020-07-23T12:33:20.483791941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.485654_99.1","section":"bucket","name":"pdivcf-16","timestamp":"2020-07-23T12:33:20.485654983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nNWx1-v33Cy8kx20mfEqCWM","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.490011_100.1","section":"bucket","name":"pdivcf-16","timestamp":"2020-07-23T12:33:20.490011274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nNWx1-v33Cy8kx20mfEqCWM","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.519518_101.1","section":"bucket.instance","name":"pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26","timestamp":"2020-07-23T12:33:20.519518015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.523662_102.1","section":"bucket.instance","name":"pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26","timestamp":"2020-07-23T12:33:20.523662515Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.526745_103.1","section":"bucket","name":"pdivcf-17","timestamp":"2020-07-23T12:33:20.526745182Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D9-gYM6k71SVeZ0yqDAlNyx","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.530907_104.1","section":"bucket","name":"pdivcf-17","timestamp":"2020-07-23T12:33:20.530907013Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D9-gYM6k71SVeZ0yqDAlNyx","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.570146_105.1","section":"bucket.instance","name":"pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27","timestamp":"2020-07-23T12:33:20.570146983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.575256_106.1","section":"bucket.instance","name":"pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27","timestamp":"2020-07-23T12:33:20.575256769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.577552_107.1","section":"bucket","name":"pdivcf-18","timestamp":"2020-07-23T12:33:20.577552876Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qfMtvuZVQROJ2UnUnTmsXWF","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.582467_108.1","section":"bucket","name":"pdivcf-18","timestamp":"2020-07-23T12:33:20.582467189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qfMtvuZVQROJ2UnUnTmsXWF","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.607611_109.1","section":"bucket.instance","name":"pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25","timestamp":"2020-07-23T12:33:20.607611890Z","data":{"read_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":1},"write_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":2},"status":{"status":"write"}}},{"id":"1_1595507600.614045_110.1","section":"bucket.instance","name":"pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25","timestamp":"2020-07-23T12:33:20.614045770Z","data":{"read_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":1},"write_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507600.619455_111.1","section":"bucket.instance","name":"pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26","timestamp":"2020-07-23T12:33:20.619455446Z","data":{"read_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":1},"write_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":2},"status":{"status":"write"}}},{"id":"1_1595507600.625710_112.1","section":"bucket.instance","name":"pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26","timestamp":"2020-07-23T12:33:20.625710444Z","data":{"read_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":1},"write_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507600.638571_113.1","section":"bucket.instance","name":"pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27","timestamp":"2020-07-23T12:33:20.638571023Z","data":{"read_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":1},"write_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":2},"status":{"status":"write"}}},{"id":"1_1595507600.647133_114.1","section":"bucket.instance","name":"pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27","timestamp":"2020-07-23T12:33:20.647133208Z","data":{"read_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":1},"write_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507704.629540_179.1","section":"bucket.instance","name":"pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28","timestamp":"2020-07-23T12:35:04.629540333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":1},"status":{"status":"write"}}},{"id":"1_1595507704.634646_185.1","section":"bucket.instance","name":"pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28","timestamp":"2020-07-23T12:35:04.634646116Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507704.636917_186.1","section":"bucket","name":"pdivcf-19","timestamp":"2020-07-23T12:35:04.636917194Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5lux1Kcalu1aSjCSsZbbPBc","ver":1},"status":{"status":"write"}}},{"id":"1_1595507704.641982_187.1","section":"bucket","name":"pdivcf-19","timestamp":"2020-07-23T12:35:04.641982741Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5lux1Kcalu1aSjCSsZbbPBc","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507704.648928_188.1","section":"bucket.instance","name":"pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28","timestamp":"2020-07-23T12:35:04.648928543Z","data":{"read_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":1},"write_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":2},"status":{"status":"write"}}},{"id":"1_1595507704.656778_189.1","section":"bucket.instance","name":"pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28","timestamp":"2020-07-23T12:35:04.656778355Z","data":{"read_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":1},"write_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507770.568084_190.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:10.568084858Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":1},"status":{"status":"write"}}},{"id":"1_1595507770.572580_207.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:10.572580901Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507770.574547_208.1","section":"bucket","name":"pdivcf-20","timestamp":"2020-07-23T12:36:10.574547102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1GieWDzPbyhzYx6dzAbvM8Y","ver":1},"status":{"status":"write"}}},{"id":"1_1595507770.579148_209.1","section":"bucket","name":"pdivcf-20","timestamp":"2020-07-23T12:36:10.579148822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1GieWDzPbyhzYx6dzAbvM8Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507790.368503_210.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:30.368503148Z","data":{"read_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":1},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":2},"status":{"status":"write"}}},{"id":"1_1595507790.379387_217.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:30.379387206Z","data":{"read_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":1},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507811.952039_218.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:51.952039133Z","data":{"read_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":2},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":3},"status":{"status":"write"}}},{"id":"1_1595507811.963376_225.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:51.963376890Z","data":{"read_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":2},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":3},"status":{"status":"complete"}}},{"id":"1_1595507833.977530_226.1","section":"bucket.instance","name":"pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30","timestamp":"2020-07-23T12:37:13.977530333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":1},"status":{"status":"write"}}},{"id":"1_1595507833.982409_233.1","section":"bucket.instance","name":"pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30","timestamp":"2020-07-23T12:37:13.982409662Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507833.984578_234.1","section":"bucket","name":"pdivcf-21","timestamp":"2020-07-23T12:37:13.984578572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxoN5QnXB8PzBQb6usiTCES","ver":1},"status":{"status":"write"}}},{"id":"1_1595507833.990265_235.1","section":"bucket","name":"pdivcf-21","timestamp":"2020-07-23T12:37:13.990265263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxoN5QnXB8PzBQb6usiTCES","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.016369_236.1","section":"bucket.instance","name":"pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31","timestamp":"2020-07-23T12:37:14.016369578Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507834.021064_237.1","section":"bucket.instance","name":"pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31","timestamp":"2020-07-23T12:37:14.021064652Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.023440_238.1","section":"bucket","name":"pdivcf-22","timestamp":"2020-07-23T12:37:14.023440018Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mOYhoW_mjO7mRP8PVQ6ZRH7","ver":1},"status":{"status":"write"}}},{"id":"1_1595507834.027916_239.1","section":"bucket","name":"pdivcf-22","timestamp":"2020-07-23T12:37:14.027916899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mOYhoW_mjO7mRP8PVQ6ZRH7","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.075022_240.1","section":"bucket.instance","name":"pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32","timestamp":"2020-07-23T12:37:14.075022229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":1},"status":{"status":"write"}}},{"id":"1_1595507834.080077_241.1","section":"bucket.instance","name":"pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32","timestamp":"2020-07-23T12:37:14.080077001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.082206_242.1","section":"bucket","name":"pdivcf-23","timestamp":"2020-07-23T12:37:14.082206069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C1euffFueI9tYW5vS7vMsgn","ver":1},"status":{"status":"write"}}},{"id":"1_1595507834.087443_243.1","section":"bucket","name":"pdivcf-23","timestamp":"2020-07-23T12:37:14.087443944Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C1euffFueI9tYW5vS7vMsgn","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.108675_244.1","section":"bucket.instance","name":"pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30","timestamp":"2020-07-23T12:37:14.108675749Z","data":{"read_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":1},"write_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":2},"status":{"status":"write"}}},{"id":"1_1595507834.113504_245.1","section":"bucket.instance","name":"pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30","timestamp":"2020-07-23T12:37:14.113504302Z","data":{"read_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":1},"write_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507834.118829_246.1","section":"bucket.instance","name":"pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31","timestamp":"2020-07-23T12:37:14.118829894Z","data":{"read_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":1},"write_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":2},"status":{"status":"write"}}},{"id":"1_1595507834.124671_247.1","section":"bucket.instance","name":"pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31","timestamp":"2020-07-23T12:37:14.124671872Z","data":{"read_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":1},"write_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507834.135668_248.1","section":"bucket.instance","name":"pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32","timestamp":"2020-07-23T12:37:14.135668915Z","data":{"read_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":1},"write_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":2},"status":{"status":"write"}}},{"id":"1_1595507834.142058_249.1","section":"bucket.instance","name":"pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32","timestamp":"2020-07-23T12:37:14.142058558Z","data":{"read_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":1},"write_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507862.301266_262.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.301266177Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.305752_275.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.305752811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.308557_276.1","section":"bucket","name":"pdivcf-24","timestamp":"2020-07-23T12:37:42.308557752Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9CmPcMhCq3DoG5OyzHLxsZp","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.313301_277.1","section":"bucket","name":"pdivcf-24","timestamp":"2020-07-23T12:37:42.313301958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9CmPcMhCq3DoG5OyzHLxsZp","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.341948_278.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.341948862Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.346312_279.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.346312178Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.348306_280.1","section":"bucket","name":"pdivcf-25","timestamp":"2020-07-23T12:37:42.348306812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_10rdqWmpSXM0gPEqJQLUt54","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.352897_281.1","section":"bucket","name":"pdivcf-25","timestamp":"2020-07-23T12:37:42.352897938Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_10rdqWmpSXM0gPEqJQLUt54","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.398608_282.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.398608049Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.402951_283.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.402951489Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.405025_284.1","section":"bucket","name":"pdivcf-26","timestamp":"2020-07-23T12:37:42.405025677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PtLBCckvoLRIiwjBffSRf4b","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.409024_285.1","section":"bucket","name":"pdivcf-26","timestamp":"2020-07-23T12:37:42.409024918Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PtLBCckvoLRIiwjBffSRf4b","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.436242_286.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.436242202Z","data":{"read_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":1},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":2},"status":{"status":"write"}}},{"id":"1_1595507862.443293_287.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.443293817Z","data":{"read_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":1},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507862.447803_288.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.447803976Z","data":{"read_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":2},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":3},"status":{"status":"write"}}},{"id":"1_1595507862.455267_289.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.455267808Z","data":{"read_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":2},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":3},"status":{"status":"complete"}}},{"id":"1_1595507862.461512_290.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.461512580Z","data":{"read_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":1},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":2},"status":{"status":"write"}}},{"id":"1_1595507862.469717_291.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.469717282Z","data":{"read_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":1},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507862.484236_292.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.484236274Z","data":{"read_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":2},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":3},"status":{"status":"write"}}},{"id":"1_1595507862.493648_293.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.493648043Z","data":{"read_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":2},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":3},"status":{"status":"complete"}}},{"id":"1_1595507862.509518_294.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.509518426Z","data":{"read_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":1},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":2},"status":{"status":"write"}}},{"id":"1_1595507862.519332_295.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.519332057Z","data":{"read_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":1},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507862.547588_296.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.547588345Z","data":{"read_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":2},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":3},"status":{"status":"write"}}},{"id":"1_1595507862.561233_297.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.561233909Z","data":{"read_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":2},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":3},"status":{"status":"complete"}}},{"id":"1_1595507882.721482_301.1","section":"bucket.instance","name":"pdivcf-27:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.36","timestamp":"2020-07-23T12:38:02.721482894Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.726106_317.1","section":"bucket.instance","name":"pdivcf-27:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.36","timestamp":"2020-07-23T12:38:02.726106004Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.728482_318.1","section":"bucket","name":"pdivcf-27","timestamp":"2020-07-23T12:38:02.728482658Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0hGeTHEMYPqYlW8vmo8ZsB4","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.733340_319.1","section":"bucket","name":"pdivcf-27","timestamp":"2020-07-23T12:38:02.733340176Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0hGeTHEMYPqYlW8vmo8ZsB4","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.764598_320.1","section":"bucket.instance","name":"pdivcf-28:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37","timestamp":"2020-07-23T12:38:02.764598242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.768937_321.1","section":"bucket.instance","name":"pdivcf-28:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37","timestamp":"2020-07-23T12:38:02.768937393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.771175_322.1","section":"bucket","name":"pdivcf-28","timestamp":"2020-07-23T12:38:02.771175166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7Rwx9HRnK8Xo_lM2eZ_spRn","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.776360_323.1","section":"bucket","name":"pdivcf-28","timestamp":"2020-07-23T12:38:02.776360587Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7Rwx9HRnK8Xo_lM2eZ_spRn","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.827591_324.1","section":"bucket.instance","name":"pdivcf-29:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38","timestamp":"2020-07-23T12:38:02.827591764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.832410_325.1","section":"bucket.instance","name":"pdivcf-29:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38","timestamp":"2020-07-23T12:38:02.832410818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.834625_326.1","section":"bucket","name":"pdivcf-29","timestamp":"2020-07-23T12:38:02.834625226Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FBpeaUcoiAUeypLntXcZ6mt","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.839185_327.1","section":"bucket","name":"pdivcf-29","timestamp":"2020-07-23T12:38:02.839185239Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FBpeaUcoiAUeypLntXcZ6mt","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.863209_328.1","section":"bucket.instance","name":"pdivcf-27:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.36","timestamp":"2020-07-23T12:38:02.863209772Z","data":{"read_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":1},"write_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":2},"status":{"status":"write"}}},{"id":"1_1595507882.870400_329.1","section":"bucket.instance","name":"pdivcf-27:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.36","timestamp":"2020-07-23T12:38:02.870400054Z","data":{"read_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":1},"write_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507882.925804_330.1","section":"bucket.instance","name":"pdivcf-28:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37","timestamp":"2020-07-23T12:38:02.925804207Z","data":{"read_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":1},"write_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":2},"status":{"status":"write"}}},{"id":"1_1595507882.936647_331.1","section":"bucket.instance","name":"pdivcf-28:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37","timestamp":"2020-07-23T12:38:02.936647924Z","data":{"read_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":1},"write_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507882.996474_332.1","section":"bucket.instance","name":"pdivcf-29:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38","timestamp":"2020-07-23T12:38:02.996474008Z","data":{"read_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":1},"write_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.007982_333.1","section":"bucket.instance","name":"pdivcf-29:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38","timestamp":"2020-07-23T12:38:03.007982412Z","data":{"read_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":1},"write_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507883.085277_334.1","section":"bucket.instance","name":"pdivcf-30:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.39","timestamp":"2020-07-23T12:38:03.085277631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.094971_335.1","section":"bucket.instance","name":"pdivcf-30:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.39","timestamp":"2020-07-23T12:38:03.094971334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.097157_336.1","section":"bucket","name":"pdivcf-30","timestamp":"2020-07-23T12:38:03.097157211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pkn26B7VC8HessJ3RMQLyfb","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.101146_337.1","section":"bucket","name":"pdivcf-30","timestamp":"2020-07-23T12:38:03.101146718Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pkn26B7VC8HessJ3RMQLyfb","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.123841_338.1","section":"bucket.instance","name":"pdivcf-31:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40","timestamp":"2020-07-23T12:38:03.123841176Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.128855_339.1","section":"bucket.instance","name":"pdivcf-31:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40","timestamp":"2020-07-23T12:38:03.128855661Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.131009_340.1","section":"bucket","name":"pdivcf-31","timestamp":"2020-07-23T12:38:03.131009569Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JpktLwZXqe4IoNIvo0sPRIm","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.135379_341.1","section":"bucket","name":"pdivcf-31","timestamp":"2020-07-23T12:38:03.135379648Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JpktLwZXqe4IoNIvo0sPRIm","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.179493_342.1","section":"bucket.instance","name":"pdivcf-32:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41","timestamp":"2020-07-23T12:38:03.179493154Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.183875_343.1","section":"bucket.instance","name":"pdivcf-32:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41","timestamp":"2020-07-23T12:38:03.183875771Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.186063_344.1","section":"bucket","name":"pdivcf-32","timestamp":"2020-07-23T12:38:03.186063685Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2yNrr1LVREKqlHu0zys3qhJ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.190110_345.1","section":"bucket","name":"pdivcf-32","timestamp":"2020-07-23T12:38:03.190110469Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2yNrr1LVREKqlHu0zys3qhJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.215750_346.1","section":"bucket.instance","name":"pdivcf-30:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.39","timestamp":"2020-07-23T12:38:03.215750124Z","data":{"read_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":1},"write_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.221277_347.1","section":"bucket.instance","name":"pdivcf-30:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.39","timestamp":"2020-07-23T12:38:03.221277172Z","data":{"read_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":1},"write_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507883.274318_348.1","section":"bucket.instance","name":"pdivcf-31:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40","timestamp":"2020-07-23T12:38:03.274318325Z","data":{"read_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":1},"write_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.286399_349.1","section":"bucket.instance","name":"pdivcf-31:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40","timestamp":"2020-07-23T12:38:03.286399254Z","data":{"read_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":1},"write_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507883.346515_350.1","section":"bucket.instance","name":"pdivcf-32:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41","timestamp":"2020-07-23T12:38:03.346515051Z","data":{"read_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":1},"write_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.356462_351.1","section":"bucket.instance","name":"pdivcf-32:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41","timestamp":"2020-07-23T12:38:03.356462902Z","data":{"read_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":1},"write_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507883.435617_352.1","section":"bucket.instance","name":"pdivcf-33:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.42","timestamp":"2020-07-23T12:38:03.435617749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.449308_353.1","section":"bucket.instance","name":"pdivcf-33:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.42","timestamp":"2020-07-23T12:38:03.449308210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.462520_354.1","section":"bucket","name":"pdivcf-33","timestamp":"2020-07-23T12:38:03.462520643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TJwU15-56F6OaBLbx9XF7mq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.466784_355.1","section":"bucket","name":"pdivcf-33","timestamp":"2020-07-23T12:38:03.466784917Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TJwU15-56F6OaBLbx9XF7mq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.564568_356.1","section":"bucket.instance","name":"pdivcf-34:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43","timestamp":"2020-07-23T12:38:03.564568818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.569408_357.1","section":"bucket.instance","name":"pdivcf-34:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43","timestamp":"2020-07-23T12:38:03.569408997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.571379_358.1","section":"bucket","name":"pdivcf-34","timestamp":"2020-07-23T12:38:03.571379967Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QaQnidDCu-eD3G2dQaeWRxh","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.575816_359.1","section":"bucket","name":"pdivcf-34","timestamp":"2020-07-23T12:38:03.575816752Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QaQnidDCu-eD3G2dQaeWRxh","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.617917_360.1","section":"bucket.instance","name":"pdivcf-35:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44","timestamp":"2020-07-23T12:38:03.617917681Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.622832_361.1","section":"bucket.instance","name":"pdivcf-35:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44","timestamp":"2020-07-23T12:38:03.622832163Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.625040_362.1","section":"bucket","name":"pdivcf-35","timestamp":"2020-07-23T12:38:03.625040094Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f8RF8Y341PAQo2j4DYcRQfj","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.630457_363.1","section":"bucket","name":"pdivcf-35","timestamp":"2020-07-23T12:38:03.630457468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f8RF8Y341PAQo2j4DYcRQfj","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.656588_364.1","section":"bucket.instance","name":"pdivcf-33:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.42","timestamp":"2020-07-23T12:38:03.656588365Z","data":{"read_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":1},"write_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.663207_365.1","section":"bucket.instance","name":"pdivcf-33:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.42","timestamp":"2020-07-23T12:38:03.663207212Z","data":{"read_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":1},"write_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507884.713429_366.1","section":"bucket.instance","name":"pdivcf-34:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43","timestamp":"2020-07-23T12:38:04.713429561Z","data":{"read_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":1},"write_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":2},"status":{"status":"write"}}},{"id":"1_1595507884.725262_367.1","section":"bucket.instance","name":"pdivcf-34:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43","timestamp":"2020-07-23T12:38:04.725262276Z","data":{"read_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":1},"write_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507885.864935_368.1","section":"bucket.instance","name":"pdivcf-35:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44","timestamp":"2020-07-23T12:38:05.864935216Z","data":{"read_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":1},"write_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":2},"status":{"status":"write"}}},{"id":"1_1595507885.876462_369.1","section":"bucket.instance","name":"pdivcf-35:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44","timestamp":"2020-07-23T12:38:05.876462797Z","data":{"read_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":1},"write_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507886.300008_370.1","section":"bucket.instance","name":"pdivcf-36:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45","timestamp":"2020-07-23T12:38:06.300008664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ps7FDMLj9P7FIJ3TQF3GFZk","ver":1},"status":{"status":"write"}}},{"id":"1_1595507886.304501_371.1","section":"bucket.instance","name":"pdivcf-36:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45","timestamp":"2020-07-23T12:38:06.304501769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ps7FDMLj9P7FIJ3TQF3GFZk","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507886.306748_372.1","section":"bucket","name":"pdivcf-36","timestamp":"2020-07-23T12:38:06.306748955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LrkNjaTY34zXjmNUserxO9w","ver":1},"status":{"status":"write"}}},{"id":"1_1595507886.311229_373.1","section":"bucket","name":"pdivcf-36","timestamp":"2020-07-23T12:38:06.311229165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LrkNjaTY34zXjmNUserxO9w","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507887.131065_374.1","section":"bucket.instance","name":"pdivcf-37:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46","timestamp":"2020-07-23T12:38:07.131065889Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Voz6xrKT44N52QQm_1jkHXq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507887.136159_375.1","section":"bucket.instance","name":"pdivcf-37:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46","timestamp":"2020-07-23T12:38:07.136159187Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Voz6xrKT44N52QQm_1jkHXq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507887.138199_376.1","section":"bucket","name":"pdivcf-37","timestamp":"2020-07-23T12:38:07.138199539Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GgAC1P5cFFfaEIRduZflSLw","ver":1},"status":{"status":"write"}}},{"id":"1_1595507887.142636_377.1","section":"bucket","name":"pdivcf-37","timestamp":"2020-07-23T12:38:07.142636002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GgAC1P5cFFfaEIRduZflSLw","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507887.514178_378.1","section":"bucket.instance","name":"pdivcf-38:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47","timestamp":"2020-07-23T12:38:07.514178897Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oY59xvqz8ppnd0uahVk4x_2","ver":1},"status":{"status":"write"}}},{"id":"1_1595507887.518533_379.1","section":"bucket.instance","name":"pdivcf-38:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47","timestamp":"2020-07-23T12:38:07.518533828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oY59xvqz8ppnd0uahVk4x_2","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507887.520752_380.1","section":"bucket","name":"pdivcf-38","timestamp":"2020-07-23T12:38:07.520752755Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N85MuH5ymUQx_bvR1jEV5oS","ver":1},"status":{"status":"write"}}},{"id":"1_1595507887.526314_381.1","section":"bucket","name":"pdivcf-38","timestamp":"2020-07-23T12:38:07.526314606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N85MuH5ymUQx_bvR1jEV5oS","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.085257_382.1","section":"bucket.instance","name":"pdivcf-39:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.48","timestamp":"2020-07-23T12:38:10.085257016Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_47jziFa-v3TzgKog19keFK3","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.090510_384.1","section":"bucket.instance","name":"pdivcf-39:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.48","timestamp":"2020-07-23T12:38:10.090510647Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_47jziFa-v3TzgKog19keFK3","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.092807_385.1","section":"bucket","name":"pdivcf-39","timestamp":"2020-07-23T12:38:10.092807212Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J6h6UEs-Kdv67K6lZ-DDqe_","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.098065_386.1","section":"bucket","name":"pdivcf-39","timestamp":"2020-07-23T12:38:10.098065606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J6h6UEs-Kdv67K6lZ-DDqe_","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.130438_387.1","section":"bucket.instance","name":"pdivcf-40:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.49","timestamp":"2020-07-23T12:38:10.130438151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jBkqIWgSeJygUbxw4jdVMVd","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.135039_388.1","section":"bucket.instance","name":"pdivcf-40:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.49","timestamp":"2020-07-23T12:38:10.135039327Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jBkqIWgSeJygUbxw4jdVMVd","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.137937_389.1","section":"bucket","name":"pdivcf-40","timestamp":"2020-07-23T12:38:10.137937297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Nsh2aQliOMNd_qa-DYEFjIr","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.146156_390.1","section":"bucket","name":"pdivcf-40","timestamp":"2020-07-23T12:38:10.146156264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Nsh2aQliOMNd_qa-DYEFjIr","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.204712_391.1","section":"bucket.instance","name":"pdivcf-41:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.50","timestamp":"2020-07-23T12:38:10.204712051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jjdhZX1cAX8QT7odbfHWP9O","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.209370_392.1","section":"bucket.instance","name":"pdivcf-41:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.50","timestamp":"2020-07-23T12:38:10.209370112Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jjdhZX1cAX8QT7odbfHWP9O","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.211763_393.1","section":"bucket","name":"pdivcf-41","timestamp":"2020-07-23T12:38:10.211763358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Voj1i0FYSZ0L7r0fMh4Tdjk","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.217061_394.1","section":"bucket","name":"pdivcf-41","timestamp":"2020-07-23T12:38:10.217061137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Voj1i0FYSZ0L7r0fMh4Tdjk","ver":1},"status":{"status":"complete"}}}] 2020-07-23T12:40:39.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '03aec9f8-1b80-4c7c-ae43-862168b0b5e6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:39.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '03aec9f8-1b80-4c7c-ae43-862168b0b5e6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:39.851 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 03aec9f8-1b80-4c7c-ae43-862168b0b5e6 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:39.894 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:39.895 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:40.147 INFO:teuthology.orchestra.run.smithi137.stdout:[{"id":"1_1595507375.318930_70.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-07-23T12:29:35.318930578Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2OfTQlQsQJ0GelLqSHyOAD_","ver":1},"status":{"status":"write"}}},{"id":"1_1595507375.329399_72.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-07-23T12:29:35.329399636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2OfTQlQsQJ0GelLqSHyOAD_","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507497.057065_74.1","section":"bucket.instance","name":"pdivcf-1:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.1","timestamp":"2020-07-23T12:31:37.057065382Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hxRQqFmO0OnO_LR85o5kUm5","ver":1},"status":{"status":"write"}}},{"id":"1_1595507497.061620_81.1","section":"bucket.instance","name":"pdivcf-1:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.1","timestamp":"2020-07-23T12:31:37.061620180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hxRQqFmO0OnO_LR85o5kUm5","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507497.063316_82.1","section":"bucket","name":"pdivcf-1","timestamp":"2020-07-23T12:31:37.063316155Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UlXS-dQKoitKY8vDjVB-gm8","ver":1},"status":{"status":"write"}}},{"id":"1_1595507497.067270_83.1","section":"bucket","name":"pdivcf-1","timestamp":"2020-07-23T12:31:37.067270716Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UlXS-dQKoitKY8vDjVB-gm8","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507497.098617_84.1","section":"bucket.instance","name":"pdivcf-2:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.2","timestamp":"2020-07-23T12:31:37.098617552Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AdQe82Xjsc-77q3kWXvfwP2","ver":1},"status":{"status":"write"}}},{"id":"1_1595507497.103291_85.1","section":"bucket.instance","name":"pdivcf-2:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.2","timestamp":"2020-07-23T12:31:37.103291974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AdQe82Xjsc-77q3kWXvfwP2","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507497.105451_86.1","section":"bucket","name":"pdivcf-2","timestamp":"2020-07-23T12:31:37.105451457Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FCKQTH1hiZp42khe9N5KHkG","ver":1},"status":{"status":"write"}}},{"id":"1_1595507497.110118_87.1","section":"bucket","name":"pdivcf-2","timestamp":"2020-07-23T12:31:37.110118928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FCKQTH1hiZp42khe9N5KHkG","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507500.591919_88.1","section":"bucket.instance","name":"pdivcf-3:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.3","timestamp":"2020-07-23T12:31:40.591919333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gQa_g97kTC2nL8PUml-gQsM","ver":1},"status":{"status":"write"}}},{"id":"1_1595507500.598942_89.1","section":"bucket.instance","name":"pdivcf-3:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.3","timestamp":"2020-07-23T12:31:40.598942621Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gQa_g97kTC2nL8PUml-gQsM","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507500.600878_90.1","section":"bucket","name":"pdivcf-3","timestamp":"2020-07-23T12:31:40.600878837Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5mXKhpLmaCwv2r1y08xfKZa","ver":1},"status":{"status":"write"}}},{"id":"1_1595507500.607977_91.1","section":"bucket","name":"pdivcf-3","timestamp":"2020-07-23T12:31:40.607977491Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5mXKhpLmaCwv2r1y08xfKZa","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.307745_92.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.4","timestamp":"2020-07-23T12:31:48.307745235Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yiZDGQu6X2F_Sk1M_XNX-aS","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.312319_95.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.4","timestamp":"2020-07-23T12:31:48.312319651Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yiZDGQu6X2F_Sk1M_XNX-aS","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.314524_96.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:48.314524642Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-Zgx0qRtoCp4uqoB0K4EqGk","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.318787_97.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:48.318787051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-Zgx0qRtoCp4uqoB0K4EqGk","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.342772_98.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.5","timestamp":"2020-07-23T12:31:48.342772112Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B4RZ8XZp_Vg_9H0AveP9SKb","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.347206_99.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.5","timestamp":"2020-07-23T12:31:48.347206077Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B4RZ8XZp_Vg_9H0AveP9SKb","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.349032_100.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:48.349032760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QSYVyw8rlDouP071b5aAg5c","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.352670_101.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:48.352670985Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QSYVyw8rlDouP071b5aAg5c","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.397942_102.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.6","timestamp":"2020-07-23T12:31:48.397942141Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Mxbry6a9nS2zmN7QABtJkqL","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.401908_103.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.6","timestamp":"2020-07-23T12:31:48.401908451Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Mxbry6a9nS2zmN7QABtJkqL","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.403730_104.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:48.403730015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nZJBxr1kdQBONurJVLbNJKj","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.407314_105.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:48.407314121Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nZJBxr1kdQBONurJVLbNJKj","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507515.909191_106.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.7","timestamp":"2020-07-23T12:31:55.909191875Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zoMNfl3WP6x69sG1A_a6fcE","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.913770_109.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.7","timestamp":"2020-07-23T12:31:55.913770150Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zoMNfl3WP6x69sG1A_a6fcE","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507515.915751_110.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:55.915751293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hU0LuGT_w40iAYldvPFrm0U","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.919252_111.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:55.919252707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hU0LuGT_w40iAYldvPFrm0U","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507515.950100_112.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.8","timestamp":"2020-07-23T12:31:55.950100507Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vltGzevaBLh8OZIhYyfm2CQ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.955276_113.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.8","timestamp":"2020-07-23T12:31:55.955276463Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vltGzevaBLh8OZIhYyfm2CQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507515.957617_114.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:55.957617299Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5r8WP0KmkR2EPhut7rsITD0","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.961660_115.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:55.961660024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5r8WP0KmkR2EPhut7rsITD0","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507515.989920_116.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.9","timestamp":"2020-07-23T12:31:55.989920273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zba4HrvvYXLNZLUZEEXyOjc","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.994394_117.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.9","timestamp":"2020-07-23T12:31:55.994394006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zba4HrvvYXLNZLUZEEXyOjc","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507515.996737_118.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:55.996737991Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XW9KXwz8j8Q2kDtlnNWtenR","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.002204_119.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.002204875Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XW9KXwz8j8Q2kDtlnNWtenR","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.038441_120.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.10","timestamp":"2020-07-23T12:31:56.038441936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yDod0geU58KV83pV2GT216-","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.042090_121.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.10","timestamp":"2020-07-23T12:31:56.042090624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yDod0geU58KV83pV2GT216-","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.044442_122.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:56.044442178Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SGxfEM8xG5yvSaheRQV2Ovd","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.047620_123.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:56.047620642Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SGxfEM8xG5yvSaheRQV2Ovd","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.098405_124.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.11","timestamp":"2020-07-23T12:31:56.098405415Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LUgRUmwvdm7gjw6CNXowxWA","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.102118_125.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.11","timestamp":"2020-07-23T12:31:56.102118377Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LUgRUmwvdm7gjw6CNXowxWA","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.103855_126.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:56.103855386Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_toR3dIX1U5fHjFxw5OD4lwp","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.106886_127.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:56.106886504Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_toR3dIX1U5fHjFxw5OD4lwp","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.198726_128.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.12","timestamp":"2020-07-23T12:31:56.198726163Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GxVDpKjDuPlGcusHQ2sPoDq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.202432_129.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.12","timestamp":"2020-07-23T12:31:56.202432137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GxVDpKjDuPlGcusHQ2sPoDq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.204287_130.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.204287823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wp9w1kaF3q9LFeeVvxwV4ZY","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.208680_131.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.208680506Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wp9w1kaF3q9LFeeVvxwV4ZY","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.310234_132.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.13","timestamp":"2020-07-23T12:31:56.310234859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pPnegWR9d5Ygh1JBGHNMLEp","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.314509_133.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.13","timestamp":"2020-07-23T12:31:56.314509529Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pPnegWR9d5Ygh1JBGHNMLEp","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.316454_134.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:56.316454484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a0pbGO-wJ9KDNLVLy0l6mVw","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.321088_135.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:56.321088724Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a0pbGO-wJ9KDNLVLy0l6mVw","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.376988_136.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.14","timestamp":"2020-07-23T12:31:56.376988037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5V1Wp4Ls0rjjCcGdYUoXxNK","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.381533_137.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.14","timestamp":"2020-07-23T12:31:56.381533072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5V1Wp4Ls0rjjCcGdYUoXxNK","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.383610_138.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:56.383610350Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qC-7doEAfyxvcolW3WYIsNi","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.387474_139.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:56.387474732Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qC-7doEAfyxvcolW3WYIsNi","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.482937_140.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.15","timestamp":"2020-07-23T12:31:56.482937943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6bW-esK6eOiMXZmLFiEt8dd","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.487703_141.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.15","timestamp":"2020-07-23T12:31:56.487703511Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6bW-esK6eOiMXZmLFiEt8dd","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.489723_142.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.489723477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RXg1UMJCNsOx1Xd-Urfc0kP","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.493752_143.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.493752921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RXg1UMJCNsOx1Xd-Urfc0kP","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507519.200871_144.1","section":"bucket.instance","name":"pdivcf-7:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.16","timestamp":"2020-07-23T12:31:59.200871054Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5kZ1JmFLTWYDEE5vSb3YN9q","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.205867_147.1","section":"bucket.instance","name":"pdivcf-7:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.16","timestamp":"2020-07-23T12:31:59.205867247Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5kZ1JmFLTWYDEE5vSb3YN9q","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.208273_148.1","section":"bucket","name":"pdivcf-7","timestamp":"2020-07-23T12:31:59.208273943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.213060_149.1","section":"bucket","name":"pdivcf-7","timestamp":"2020-07-23T12:31:59.213060668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.236100_150.1","section":"bucket.instance","name":"pdivcf-8:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.17","timestamp":"2020-07-23T12:31:59.236100590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tCGnu2GnZ4klKs-yTkLNnbJ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.242796_151.1","section":"bucket.instance","name":"pdivcf-8:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.17","timestamp":"2020-07-23T12:31:59.242796192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tCGnu2GnZ4klKs-yTkLNnbJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.246769_152.1","section":"bucket","name":"pdivcf-8","timestamp":"2020-07-23T12:31:59.246769828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.253139_153.1","section":"bucket","name":"pdivcf-8","timestamp":"2020-07-23T12:31:59.253139879Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.306952_154.1","section":"bucket.instance","name":"pdivcf-9:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.18","timestamp":"2020-07-23T12:31:59.306952465Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kbWQDeLkSgvmeNGJdVE78Au","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.312907_155.1","section":"bucket.instance","name":"pdivcf-9:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.18","timestamp":"2020-07-23T12:31:59.312907044Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kbWQDeLkSgvmeNGJdVE78Au","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.315224_156.1","section":"bucket","name":"pdivcf-9","timestamp":"2020-07-23T12:31:59.315224409Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.320666_157.1","section":"bucket","name":"pdivcf-9","timestamp":"2020-07-23T12:31:59.320666771Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507528.014327_158.1","section":"bucket","name":"pdivcf-7","timestamp":"2020-07-23T12:32:08.014327176Z","data":{"read_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":1},"write_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":2},"status":{"status":"remove"}}},{"id":"1_1595507528.020271_161.1","section":"bucket","name":"pdivcf-7","timestamp":"2020-07-23T12:32:08.020271414Z","data":{"read_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":1},"write_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507528.077441_162.1","section":"bucket","name":"pdivcf-8","timestamp":"2020-07-23T12:32:08.077441314Z","data":{"read_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":1},"write_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":2},"status":{"status":"remove"}}},{"id":"1_1595507528.083398_163.1","section":"bucket","name":"pdivcf-8","timestamp":"2020-07-23T12:32:08.083398771Z","data":{"read_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":1},"write_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507528.169614_164.1","section":"bucket","name":"pdivcf-9","timestamp":"2020-07-23T12:32:08.169614550Z","data":{"read_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":1},"write_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":2},"status":{"status":"remove"}}},{"id":"1_1595507528.176421_165.1","section":"bucket","name":"pdivcf-9","timestamp":"2020-07-23T12:32:08.176421339Z","data":{"read_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":1},"write_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507537.173491_46.1","section":"bucket.instance","name":"pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19","timestamp":"2020-07-23T12:32:17.173491511Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N7MWXq20eARG4bplabEWJWI","ver":1},"status":{"status":"write"}}},{"id":"1_1595507537.178224_55.1","section":"bucket.instance","name":"pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19","timestamp":"2020-07-23T12:32:17.178224061Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N7MWXq20eARG4bplabEWJWI","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507537.180408_56.1","section":"bucket","name":"pdivcf-10","timestamp":"2020-07-23T12:32:17.180408142Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eWs9kH85eL7ExiBJ3M7eDWt","ver":1},"status":{"status":"write"}}},{"id":"1_1595507537.185066_57.1","section":"bucket","name":"pdivcf-10","timestamp":"2020-07-23T12:32:17.185066160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eWs9kH85eL7ExiBJ3M7eDWt","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507538.116775_58.1","section":"bucket.instance","name":"pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20","timestamp":"2020-07-23T12:32:18.116775021Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0d0yAZmihVZhaW7NV6g_3kX","ver":1},"status":{"status":"write"}}},{"id":"1_1595507538.121315_59.1","section":"bucket.instance","name":"pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20","timestamp":"2020-07-23T12:32:18.121315724Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0d0yAZmihVZhaW7NV6g_3kX","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507538.123177_60.1","section":"bucket","name":"pdivcf-11","timestamp":"2020-07-23T12:32:18.123177285Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E1SDCqpJIo2DB_cVYeRt5Fj","ver":1},"status":{"status":"write"}}},{"id":"1_1595507538.130212_61.1","section":"bucket","name":"pdivcf-11","timestamp":"2020-07-23T12:32:18.130212887Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E1SDCqpJIo2DB_cVYeRt5Fj","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507538.842232_62.1","section":"bucket.instance","name":"pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21","timestamp":"2020-07-23T12:32:18.842232990Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DSr1_h9uAWM0Aupnq07MG-T","ver":1},"status":{"status":"write"}}},{"id":"1_1595507538.846555_63.1","section":"bucket.instance","name":"pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21","timestamp":"2020-07-23T12:32:18.846555026Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DSr1_h9uAWM0Aupnq07MG-T","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507538.848627_64.1","section":"bucket","name":"pdivcf-12","timestamp":"2020-07-23T12:32:18.848627622Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Idw7iMdTOFPnkuR0PvQ7oMk","ver":1},"status":{"status":"write"}}},{"id":"1_1595507538.853263_65.1","section":"bucket","name":"pdivcf-12","timestamp":"2020-07-23T12:32:18.853263990Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Idw7iMdTOFPnkuR0PvQ7oMk","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.772196_66.1","section":"bucket.instance","name":"pdivcf-13:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22","timestamp":"2020-07-23T12:32:37.772196573Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Na-elmOmjZsaFeM-ChUYR-E","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.776481_73.1","section":"bucket.instance","name":"pdivcf-13:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22","timestamp":"2020-07-23T12:32:37.776481061Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Na-elmOmjZsaFeM-ChUYR-E","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.778958_74.1","section":"bucket","name":"pdivcf-13","timestamp":"2020-07-23T12:32:37.778958954Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WAdUfnwKp-iQ-htLlTSCyaD","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.787127_75.1","section":"bucket","name":"pdivcf-13","timestamp":"2020-07-23T12:32:37.787127121Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WAdUfnwKp-iQ-htLlTSCyaD","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.815884_76.1","section":"bucket.instance","name":"pdivcf-14:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23","timestamp":"2020-07-23T12:32:37.815884493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BerFUu6TepMUW1OCfDzLt1L","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.820651_77.1","section":"bucket.instance","name":"pdivcf-14:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23","timestamp":"2020-07-23T12:32:37.820651075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BerFUu6TepMUW1OCfDzLt1L","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.823023_78.1","section":"bucket","name":"pdivcf-14","timestamp":"2020-07-23T12:32:37.823023440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBn5niBkGEfAMwLcYB5EkPd","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.828030_79.1","section":"bucket","name":"pdivcf-14","timestamp":"2020-07-23T12:32:37.828030042Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBn5niBkGEfAMwLcYB5EkPd","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.877028_80.1","section":"bucket.instance","name":"pdivcf-15:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24","timestamp":"2020-07-23T12:32:37.877028266Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ph9awnaTrqKViZtfefg5dn8","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.881682_81.1","section":"bucket.instance","name":"pdivcf-15:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24","timestamp":"2020-07-23T12:32:37.881682539Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ph9awnaTrqKViZtfefg5dn8","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.884047_82.1","section":"bucket","name":"pdivcf-15","timestamp":"2020-07-23T12:32:37.884047505Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HaDbmpKCs0MvAJEp019L9md","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.889012_83.1","section":"bucket","name":"pdivcf-15","timestamp":"2020-07-23T12:32:37.889012624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HaDbmpKCs0MvAJEp019L9md","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.476318_84.1","section":"bucket.instance","name":"pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25","timestamp":"2020-07-23T12:33:20.476318852Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.483791_98.1","section":"bucket.instance","name":"pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25","timestamp":"2020-07-23T12:33:20.483791941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.485654_99.1","section":"bucket","name":"pdivcf-16","timestamp":"2020-07-23T12:33:20.485654983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nNWx1-v33Cy8kx20mfEqCWM","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.490011_100.1","section":"bucket","name":"pdivcf-16","timestamp":"2020-07-23T12:33:20.490011274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nNWx1-v33Cy8kx20mfEqCWM","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.519518_101.1","section":"bucket.instance","name":"pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26","timestamp":"2020-07-23T12:33:20.519518015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.523662_102.1","section":"bucket.instance","name":"pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26","timestamp":"2020-07-23T12:33:20.523662515Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.526745_103.1","section":"bucket","name":"pdivcf-17","timestamp":"2020-07-23T12:33:20.526745182Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D9-gYM6k71SVeZ0yqDAlNyx","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.530907_104.1","section":"bucket","name":"pdivcf-17","timestamp":"2020-07-23T12:33:20.530907013Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D9-gYM6k71SVeZ0yqDAlNyx","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.570146_105.1","section":"bucket.instance","name":"pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27","timestamp":"2020-07-23T12:33:20.570146983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.575256_106.1","section":"bucket.instance","name":"pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27","timestamp":"2020-07-23T12:33:20.575256769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.577552_107.1","section":"bucket","name":"pdivcf-18","timestamp":"2020-07-23T12:33:20.577552876Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qfMtvuZVQROJ2UnUnTmsXWF","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.582467_108.1","section":"bucket","name":"pdivcf-18","timestamp":"2020-07-23T12:33:20.582467189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qfMtvuZVQROJ2UnUnTmsXWF","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.607611_109.1","section":"bucket.instance","name":"pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25","timestamp":"2020-07-23T12:33:20.607611890Z","data":{"read_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":1},"write_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":2},"status":{"status":"write"}}},{"id":"1_1595507600.614045_110.1","section":"bucket.instance","name":"pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25","timestamp":"2020-07-23T12:33:20.614045770Z","data":{"read_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":1},"write_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507600.619455_111.1","section":"bucket.instance","name":"pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26","timestamp":"2020-07-23T12:33:20.619455446Z","data":{"read_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":1},"write_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":2},"status":{"status":"write"}}},{"id":"1_1595507600.625710_112.1","section":"bucket.instance","name":"pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26","timestamp":"2020-07-23T12:33:20.625710444Z","data":{"read_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":1},"write_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507600.638571_113.1","section":"bucket.instance","name":"pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27","timestamp":"2020-07-23T12:33:20.638571023Z","data":{"read_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":1},"write_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":2},"status":{"status":"write"}}},{"id":"1_1595507600.647133_114.1","section":"bucket.instance","name":"pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27","timestamp":"2020-07-23T12:33:20.647133208Z","data":{"read_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":1},"write_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507704.629540_179.1","section":"bucket.instance","name":"pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28","timestamp":"2020-07-23T12:35:04.629540333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":1},"status":{"status":"write"}}},{"id":"1_1595507704.634646_185.1","section":"bucket.instance","name":"pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28","timestamp":"2020-07-23T12:35:04.634646116Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507704.636917_186.1","section":"bucket","name":"pdivcf-19","timestamp":"2020-07-23T12:35:04.636917194Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5lux1Kcalu1aSjCSsZbbPBc","ver":1},"status":{"status":"write"}}},{"id":"1_1595507704.641982_187.1","section":"bucket","name":"pdivcf-19","timestamp":"2020-07-23T12:35:04.641982741Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5lux1Kcalu1aSjCSsZbbPBc","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507704.648928_188.1","section":"bucket.instance","name":"pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28","timestamp":"2020-07-23T12:35:04.648928543Z","data":{"read_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":1},"write_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":2},"status":{"status":"write"}}},{"id":"1_1595507704.656778_189.1","section":"bucket.instance","name":"pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28","timestamp":"2020-07-23T12:35:04.656778355Z","data":{"read_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":1},"write_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507770.568084_190.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:10.568084858Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":1},"status":{"status":"write"}}},{"id":"1_1595507770.572580_207.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:10.572580901Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507770.574547_208.1","section":"bucket","name":"pdivcf-20","timestamp":"2020-07-23T12:36:10.574547102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1GieWDzPbyhzYx6dzAbvM8Y","ver":1},"status":{"status":"write"}}},{"id":"1_1595507770.579148_209.1","section":"bucket","name":"pdivcf-20","timestamp":"2020-07-23T12:36:10.579148822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1GieWDzPbyhzYx6dzAbvM8Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507790.368503_210.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:30.368503148Z","data":{"read_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":1},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":2},"status":{"status":"write"}}},{"id":"1_1595507790.379387_217.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:30.379387206Z","data":{"read_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":1},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507811.952039_218.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:51.952039133Z","data":{"read_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":2},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":3},"status":{"status":"write"}}},{"id":"1_1595507811.963376_225.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:51.963376890Z","data":{"read_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":2},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":3},"status":{"status":"complete"}}},{"id":"1_1595507833.977530_226.1","section":"bucket.instance","name":"pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30","timestamp":"2020-07-23T12:37:13.977530333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":1},"status":{"status":"write"}}},{"id":"1_1595507833.982409_233.1","section":"bucket.instance","name":"pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30","timestamp":"2020-07-23T12:37:13.982409662Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507833.984578_234.1","section":"bucket","name":"pdivcf-21","timestamp":"2020-07-23T12:37:13.984578572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxoN5QnXB8PzBQb6usiTCES","ver":1},"status":{"status":"write"}}},{"id":"1_1595507833.990265_235.1","section":"bucket","name":"pdivcf-21","timestamp":"2020-07-23T12:37:13.990265263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxoN5QnXB8PzBQb6usiTCES","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.016369_236.1","section":"bucket.instance","name":"pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31","timestamp":"2020-07-23T12:37:14.016369578Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507834.021064_237.1","section":"bucket.instance","name":"pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31","timestamp":"2020-07-23T12:37:14.021064652Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.023440_238.1","section":"bucket","name":"pdivcf-22","timestamp":"2020-07-23T12:37:14.023440018Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mOYhoW_mjO7mRP8PVQ6ZRH7","ver":1},"status":{"status":"write"}}},{"id":"1_1595507834.027916_239.1","section":"bucket","name":"pdivcf-22","timestamp":"2020-07-23T12:37:14.027916899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mOYhoW_mjO7mRP8PVQ6ZRH7","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.075022_240.1","section":"bucket.instance","name":"pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32","timestamp":"2020-07-23T12:37:14.075022229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":1},"status":{"status":"write"}}},{"id":"1_1595507834.080077_241.1","section":"bucket.instance","name":"pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32","timestamp":"2020-07-23T12:37:14.080077001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.082206_242.1","section":"bucket","name":"pdivcf-23","timestamp":"2020-07-23T12:37:14.082206069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C1euffFueI9tYW5vS7vMsgn","ver":1},"status":{"status":"write"}}},{"id":"1_1595507834.087443_243.1","section":"bucket","name":"pdivcf-23","timestamp":"2020-07-23T12:37:14.087443944Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C1euffFueI9tYW5vS7vMsgn","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.108675_244.1","section":"bucket.instance","name":"pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30","timestamp":"2020-07-23T12:37:14.108675749Z","data":{"read_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":1},"write_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":2},"status":{"status":"write"}}},{"id":"1_1595507834.113504_245.1","section":"bucket.instance","name":"pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30","timestamp":"2020-07-23T12:37:14.113504302Z","data":{"read_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":1},"write_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507834.118829_246.1","section":"bucket.instance","name":"pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31","timestamp":"2020-07-23T12:37:14.118829894Z","data":{"read_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":1},"write_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":2},"status":{"status":"write"}}},{"id":"1_1595507834.124671_247.1","section":"bucket.instance","name":"pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31","timestamp":"2020-07-23T12:37:14.124671872Z","data":{"read_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":1},"write_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507834.135668_248.1","section":"bucket.instance","name":"pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32","timestamp":"2020-07-23T12:37:14.135668915Z","data":{"read_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":1},"write_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":2},"status":{"status":"write"}}},{"id":"1_1595507834.142058_249.1","section":"bucket.instance","name":"pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32","timestamp":"2020-07-23T12:37:14.142058558Z","data":{"read_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":1},"write_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507862.301266_262.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.301266177Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.305752_275.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.305752811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.308557_276.1","section":"bucket","name":"pdivcf-24","timestamp":"2020-07-23T12:37:42.308557752Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9CmPcMhCq3DoG5OyzHLxsZp","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.313301_277.1","section":"bucket","name":"pdivcf-24","timestamp":"2020-07-23T12:37:42.313301958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9CmPcMhCq3DoG5OyzHLxsZp","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.341948_278.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.341948862Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.346312_279.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.346312178Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.348306_280.1","section":"bucket","name":"pdivcf-25","timestamp":"2020-07-23T12:37:42.348306812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_10rdqWmpSXM0gPEqJQLUt54","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.352897_281.1","section":"bucket","name":"pdivcf-25","timestamp":"2020-07-23T12:37:42.352897938Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_10rdqWmpSXM0gPEqJQLUt54","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.398608_282.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.398608049Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.402951_283.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.402951489Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.405025_284.1","section":"bucket","name":"pdivcf-26","timestamp":"2020-07-23T12:37:42.405025677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PtLBCckvoLRIiwjBffSRf4b","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.409024_285.1","section":"bucket","name":"pdivcf-26","timestamp":"2020-07-23T12:37:42.409024918Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PtLBCckvoLRIiwjBffSRf4b","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.436242_286.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.436242202Z","data":{"read_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":1},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":2},"status":{"status":"write"}}},{"id":"1_1595507862.443293_287.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.443293817Z","data":{"read_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":1},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507862.447803_288.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.447803976Z","data":{"read_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":2},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":3},"status":{"status":"write"}}},{"id":"1_1595507862.455267_289.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.455267808Z","data":{"read_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":2},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":3},"status":{"status":"complete"}}},{"id":"1_1595507862.461512_290.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.461512580Z","data":{"read_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":1},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":2},"status":{"status":"write"}}},{"id":"1_1595507862.469717_291.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.469717282Z","data":{"read_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":1},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507862.484236_292.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.484236274Z","data":{"read_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":2},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":3},"status":{"status":"write"}}},{"id":"1_1595507862.493648_293.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.493648043Z","data":{"read_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":2},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":3},"status":{"status":"complete"}}},{"id":"1_1595507862.509518_294.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.509518426Z","data":{"read_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":1},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":2},"status":{"status":"write"}}},{"id":"1_1595507862.519332_295.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.519332057Z","data":{"read_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":1},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507862.547588_296.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.547588345Z","data":{"read_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":2},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":3},"status":{"status":"write"}}},{"id":"1_1595507862.561233_297.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.561233909Z","data":{"read_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":2},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":3},"status":{"status":"complete"}}},{"id":"1_1595507882.721482_301.1","section":"bucket.instance","name":"pdivcf-27:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.36","timestamp":"2020-07-23T12:38:02.721482894Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.726106_317.1","section":"bucket.instance","name":"pdivcf-27:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.36","timestamp":"2020-07-23T12:38:02.726106004Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.728482_318.1","section":"bucket","name":"pdivcf-27","timestamp":"2020-07-23T12:38:02.728482658Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0hGeTHEMYPqYlW8vmo8ZsB4","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.733340_319.1","section":"bucket","name":"pdivcf-27","timestamp":"2020-07-23T12:38:02.733340176Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0hGeTHEMYPqYlW8vmo8ZsB4","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.764598_320.1","section":"bucket.instance","name":"pdivcf-28:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37","timestamp":"2020-07-23T12:38:02.764598242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.768937_321.1","section":"bucket.instance","name":"pdivcf-28:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37","timestamp":"2020-07-23T12:38:02.768937393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.771175_322.1","section":"bucket","name":"pdivcf-28","timestamp":"2020-07-23T12:38:02.771175166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7Rwx9HRnK8Xo_lM2eZ_spRn","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.776360_323.1","section":"bucket","name":"pdivcf-28","timestamp":"2020-07-23T12:38:02.776360587Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7Rwx9HRnK8Xo_lM2eZ_spRn","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.827591_324.1","section":"bucket.instance","name":"pdivcf-29:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38","timestamp":"2020-07-23T12:38:02.827591764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.832410_325.1","section":"bucket.instance","name":"pdivcf-29:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38","timestamp":"2020-07-23T12:38:02.832410818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.834625_326.1","section":"bucket","name":"pdivcf-29","timestamp":"2020-07-23T12:38:02.834625226Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FBpeaUcoiAUeypLntXcZ6mt","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.839185_327.1","section":"bucket","name":"pdivcf-29","timestamp":"2020-07-23T12:38:02.839185239Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FBpeaUcoiAUeypLntXcZ6mt","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.863209_328.1","section":"bucket.instance","name":"pdivcf-27:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.36","timestamp":"2020-07-23T12:38:02.863209772Z","data":{"read_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":1},"write_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":2},"status":{"status":"write"}}},{"id":"1_1595507882.870400_329.1","section":"bucket.instance","name":"pdivcf-27:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.36","timestamp":"2020-07-23T12:38:02.870400054Z","data":{"read_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":1},"write_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507882.925804_330.1","section":"bucket.instance","name":"pdivcf-28:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37","timestamp":"2020-07-23T12:38:02.925804207Z","data":{"read_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":1},"write_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":2},"status":{"status":"write"}}},{"id":"1_1595507882.936647_331.1","section":"bucket.instance","name":"pdivcf-28:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37","timestamp":"2020-07-23T12:38:02.936647924Z","data":{"read_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":1},"write_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507882.996474_332.1","section":"bucket.instance","name":"pdivcf-29:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38","timestamp":"2020-07-23T12:38:02.996474008Z","data":{"read_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":1},"write_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.007982_333.1","section":"bucket.instance","name":"pdivcf-29:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38","timestamp":"2020-07-23T12:38:03.007982412Z","data":{"read_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":1},"write_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507883.085277_334.1","section":"bucket.instance","name":"pdivcf-30:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.39","timestamp":"2020-07-23T12:38:03.085277631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.094971_335.1","section":"bucket.instance","name":"pdivcf-30:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.39","timestamp":"2020-07-23T12:38:03.094971334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.097157_336.1","section":"bucket","name":"pdivcf-30","timestamp":"2020-07-23T12:38:03.097157211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pkn26B7VC8HessJ3RMQLyfb","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.101146_337.1","section":"bucket","name":"pdivcf-30","timestamp":"2020-07-23T12:38:03.101146718Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pkn26B7VC8HessJ3RMQLyfb","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.123841_338.1","section":"bucket.instance","name":"pdivcf-31:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40","timestamp":"2020-07-23T12:38:03.123841176Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.128855_339.1","section":"bucket.instance","name":"pdivcf-31:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40","timestamp":"2020-07-23T12:38:03.128855661Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.131009_340.1","section":"bucket","name":"pdivcf-31","timestamp":"2020-07-23T12:38:03.131009569Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JpktLwZXqe4IoNIvo0sPRIm","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.135379_341.1","section":"bucket","name":"pdivcf-31","timestamp":"2020-07-23T12:38:03.135379648Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JpktLwZXqe4IoNIvo0sPRIm","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.179493_342.1","section":"bucket.instance","name":"pdivcf-32:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41","timestamp":"2020-07-23T12:38:03.179493154Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.183875_343.1","section":"bucket.instance","name":"pdivcf-32:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41","timestamp":"2020-07-23T12:38:03.183875771Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.186063_344.1","section":"bucket","name":"pdivcf-32","timestamp":"2020-07-23T12:38:03.186063685Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2yNrr1LVREKqlHu0zys3qhJ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.190110_345.1","section":"bucket","name":"pdivcf-32","timestamp":"2020-07-23T12:38:03.190110469Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2yNrr1LVREKqlHu0zys3qhJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.215750_346.1","section":"bucket.instance","name":"pdivcf-30:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.39","timestamp":"2020-07-23T12:38:03.215750124Z","data":{"read_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":1},"write_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.221277_347.1","section":"bucket.instance","name":"pdivcf-30:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.39","timestamp":"2020-07-23T12:38:03.221277172Z","data":{"read_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":1},"write_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507883.274318_348.1","section":"bucket.instance","name":"pdivcf-31:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40","timestamp":"2020-07-23T12:38:03.274318325Z","data":{"read_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":1},"write_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.286399_349.1","section":"bucket.instance","name":"pdivcf-31:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40","timestamp":"2020-07-23T12:38:03.286399254Z","data":{"read_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":1},"write_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507883.346515_350.1","section":"bucket.instance","name":"pdivcf-32:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41","timestamp":"2020-07-23T12:38:03.346515051Z","data":{"read_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":1},"write_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.356462_351.1","section":"bucket.instance","name":"pdivcf-32:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41","timestamp":"2020-07-23T12:38:03.356462902Z","data":{"read_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":1},"write_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507883.435617_352.1","section":"bucket.instance","name":"pdivcf-33:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.42","timestamp":"2020-07-23T12:38:03.435617749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.449308_353.1","section":"bucket.instance","name":"pdivcf-33:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.42","timestamp":"2020-07-23T12:38:03.449308210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.462520_354.1","section":"bucket","name":"pdivcf-33","timestamp":"2020-07-23T12:38:03.462520643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TJwU15-56F6OaBLbx9XF7mq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.466784_355.1","section":"bucket","name":"pdivcf-33","timestamp":"2020-07-23T12:38:03.466784917Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TJwU15-56F6OaBLbx9XF7mq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.564568_356.1","section":"bucket.instance","name":"pdivcf-34:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43","timestamp":"2020-07-23T12:38:03.564568818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.569408_357.1","section":"bucket.instance","name":"pdivcf-34:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43","timestamp":"2020-07-23T12:38:03.569408997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.571379_358.1","section":"bucket","name":"pdivcf-34","timestamp":"2020-07-23T12:38:03.571379967Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QaQnidDCu-eD3G2dQaeWRxh","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.575816_359.1","section":"bucket","name":"pdivcf-34","timestamp":"2020-07-23T12:38:03.575816752Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QaQnidDCu-eD3G2dQaeWRxh","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.617917_360.1","section":"bucket.instance","name":"pdivcf-35:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44","timestamp":"2020-07-23T12:38:03.617917681Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.622832_361.1","section":"bucket.instance","name":"pdivcf-35:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44","timestamp":"2020-07-23T12:38:03.622832163Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.625040_362.1","section":"bucket","name":"pdivcf-35","timestamp":"2020-07-23T12:38:03.625040094Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f8RF8Y341PAQo2j4DYcRQfj","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.630457_363.1","section":"bucket","name":"pdivcf-35","timestamp":"2020-07-23T12:38:03.630457468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f8RF8Y341PAQo2j4DYcRQfj","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.656588_364.1","section":"bucket.instance","name":"pdivcf-33:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.42","timestamp":"2020-07-23T12:38:03.656588365Z","data":{"read_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":1},"write_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.663207_365.1","section":"bucket.instance","name":"pdivcf-33:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.42","timestamp":"2020-07-23T12:38:03.663207212Z","data":{"read_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":1},"write_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507884.713429_366.1","section":"bucket.instance","name":"pdivcf-34:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43","timestamp":"2020-07-23T12:38:04.713429561Z","data":{"read_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":1},"write_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":2},"status":{"status":"write"}}},{"id":"1_1595507884.725262_367.1","section":"bucket.instance","name":"pdivcf-34:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43","timestamp":"2020-07-23T12:38:04.725262276Z","data":{"read_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":1},"write_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507885.864935_368.1","section":"bucket.instance","name":"pdivcf-35:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44","timestamp":"2020-07-23T12:38:05.864935216Z","data":{"read_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":1},"write_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":2},"status":{"status":"write"}}},{"id":"1_1595507885.876462_369.1","section":"bucket.instance","name":"pdivcf-35:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44","timestamp":"2020-07-23T12:38:05.876462797Z","data":{"read_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":1},"write_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507886.300008_370.1","section":"bucket.instance","name":"pdivcf-36:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45","timestamp":"2020-07-23T12:38:06.300008664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ps7FDMLj9P7FIJ3TQF3GFZk","ver":1},"status":{"status":"write"}}},{"id":"1_1595507886.304501_371.1","section":"bucket.instance","name":"pdivcf-36:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45","timestamp":"2020-07-23T12:38:06.304501769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ps7FDMLj9P7FIJ3TQF3GFZk","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507886.306748_372.1","section":"bucket","name":"pdivcf-36","timestamp":"2020-07-23T12:38:06.306748955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LrkNjaTY34zXjmNUserxO9w","ver":1},"status":{"status":"write"}}},{"id":"1_1595507886.311229_373.1","section":"bucket","name":"pdivcf-36","timestamp":"2020-07-23T12:38:06.311229165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LrkNjaTY34zXjmNUserxO9w","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507887.131065_374.1","section":"bucket.instance","name":"pdivcf-37:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46","timestamp":"2020-07-23T12:38:07.131065889Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Voz6xrKT44N52QQm_1jkHXq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507887.136159_375.1","section":"bucket.instance","name":"pdivcf-37:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46","timestamp":"2020-07-23T12:38:07.136159187Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Voz6xrKT44N52QQm_1jkHXq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507887.138199_376.1","section":"bucket","name":"pdivcf-37","timestamp":"2020-07-23T12:38:07.138199539Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GgAC1P5cFFfaEIRduZflSLw","ver":1},"status":{"status":"write"}}},{"id":"1_1595507887.142636_377.1","section":"bucket","name":"pdivcf-37","timestamp":"2020-07-23T12:38:07.142636002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GgAC1P5cFFfaEIRduZflSLw","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507887.514178_378.1","section":"bucket.instance","name":"pdivcf-38:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47","timestamp":"2020-07-23T12:38:07.514178897Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oY59xvqz8ppnd0uahVk4x_2","ver":1},"status":{"status":"write"}}},{"id":"1_1595507887.518533_379.1","section":"bucket.instance","name":"pdivcf-38:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47","timestamp":"2020-07-23T12:38:07.518533828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oY59xvqz8ppnd0uahVk4x_2","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507887.520752_380.1","section":"bucket","name":"pdivcf-38","timestamp":"2020-07-23T12:38:07.520752755Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N85MuH5ymUQx_bvR1jEV5oS","ver":1},"status":{"status":"write"}}},{"id":"1_1595507887.526314_381.1","section":"bucket","name":"pdivcf-38","timestamp":"2020-07-23T12:38:07.526314606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N85MuH5ymUQx_bvR1jEV5oS","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.085257_382.1","section":"bucket.instance","name":"pdivcf-39:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.48","timestamp":"2020-07-23T12:38:10.085257016Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_47jziFa-v3TzgKog19keFK3","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.090510_384.1","section":"bucket.instance","name":"pdivcf-39:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.48","timestamp":"2020-07-23T12:38:10.090510647Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_47jziFa-v3TzgKog19keFK3","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.092807_385.1","section":"bucket","name":"pdivcf-39","timestamp":"2020-07-23T12:38:10.092807212Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J6h6UEs-Kdv67K6lZ-DDqe_","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.098065_386.1","section":"bucket","name":"pdivcf-39","timestamp":"2020-07-23T12:38:10.098065606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J6h6UEs-Kdv67K6lZ-DDqe_","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.130438_387.1","section":"bucket.instance","name":"pdivcf-40:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.49","timestamp":"2020-07-23T12:38:10.130438151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jBkqIWgSeJygUbxw4jdVMVd","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.135039_388.1","section":"bucket.instance","name":"pdivcf-40:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.49","timestamp":"2020-07-23T12:38:10.135039327Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jBkqIWgSeJygUbxw4jdVMVd","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.137937_389.1","section":"bucket","name":"pdivcf-40","timestamp":"2020-07-23T12:38:10.137937297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Nsh2aQliOMNd_qa-DYEFjIr","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.146156_390.1","section":"bucket","name":"pdivcf-40","timestamp":"2020-07-23T12:38:10.146156264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Nsh2aQliOMNd_qa-DYEFjIr","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.204712_391.1","section":"bucket.instance","name":"pdivcf-41:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.50","timestamp":"2020-07-23T12:38:10.204712051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jjdhZX1cAX8QT7odbfHWP9O","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.209370_392.1","section":"bucket.instance","name":"pdivcf-41:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.50","timestamp":"2020-07-23T12:38:10.209370112Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jjdhZX1cAX8QT7odbfHWP9O","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.211763_393.1","section":"bucket","name":"pdivcf-41","timestamp":"2020-07-23T12:38:10.211763358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Voj1i0FYSZ0L7r0fMh4Tdjk","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.217061_394.1","section":"bucket","name":"pdivcf-41","timestamp":"2020-07-23T12:38:10.217061137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Voj1i0FYSZ0L7r0fMh4Tdjk","ver":1},"status":{"status":"complete"}}}] 2020-07-23T12:40:40.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '03aec9f8-1b80-4c7c-ae43-862168b0b5e6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:40.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '03aec9f8-1b80-4c7c-ae43-862168b0b5e6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:40.151 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 03aec9f8-1b80-4c7c-ae43-862168b0b5e6 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:40.196 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:40.196 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:40.468 INFO:teuthology.orchestra.run.smithi133.stdout:[{"id":"1_1595507375.318930_70.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-07-23T12:29:35.318930578Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2OfTQlQsQJ0GelLqSHyOAD_","ver":1},"status":{"status":"write"}}},{"id":"1_1595507375.329399_72.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-07-23T12:29:35.329399636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2OfTQlQsQJ0GelLqSHyOAD_","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507497.057065_74.1","section":"bucket.instance","name":"pdivcf-1:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.1","timestamp":"2020-07-23T12:31:37.057065382Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hxRQqFmO0OnO_LR85o5kUm5","ver":1},"status":{"status":"write"}}},{"id":"1_1595507497.061620_81.1","section":"bucket.instance","name":"pdivcf-1:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.1","timestamp":"2020-07-23T12:31:37.061620180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hxRQqFmO0OnO_LR85o5kUm5","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507497.063316_82.1","section":"bucket","name":"pdivcf-1","timestamp":"2020-07-23T12:31:37.063316155Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UlXS-dQKoitKY8vDjVB-gm8","ver":1},"status":{"status":"write"}}},{"id":"1_1595507497.067270_83.1","section":"bucket","name":"pdivcf-1","timestamp":"2020-07-23T12:31:37.067270716Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UlXS-dQKoitKY8vDjVB-gm8","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507497.098617_84.1","section":"bucket.instance","name":"pdivcf-2:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.2","timestamp":"2020-07-23T12:31:37.098617552Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AdQe82Xjsc-77q3kWXvfwP2","ver":1},"status":{"status":"write"}}},{"id":"1_1595507497.103291_85.1","section":"bucket.instance","name":"pdivcf-2:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.2","timestamp":"2020-07-23T12:31:37.103291974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AdQe82Xjsc-77q3kWXvfwP2","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507497.105451_86.1","section":"bucket","name":"pdivcf-2","timestamp":"2020-07-23T12:31:37.105451457Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FCKQTH1hiZp42khe9N5KHkG","ver":1},"status":{"status":"write"}}},{"id":"1_1595507497.110118_87.1","section":"bucket","name":"pdivcf-2","timestamp":"2020-07-23T12:31:37.110118928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FCKQTH1hiZp42khe9N5KHkG","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507500.591919_88.1","section":"bucket.instance","name":"pdivcf-3:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.3","timestamp":"2020-07-23T12:31:40.591919333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gQa_g97kTC2nL8PUml-gQsM","ver":1},"status":{"status":"write"}}},{"id":"1_1595507500.598942_89.1","section":"bucket.instance","name":"pdivcf-3:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.3","timestamp":"2020-07-23T12:31:40.598942621Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gQa_g97kTC2nL8PUml-gQsM","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507500.600878_90.1","section":"bucket","name":"pdivcf-3","timestamp":"2020-07-23T12:31:40.600878837Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5mXKhpLmaCwv2r1y08xfKZa","ver":1},"status":{"status":"write"}}},{"id":"1_1595507500.607977_91.1","section":"bucket","name":"pdivcf-3","timestamp":"2020-07-23T12:31:40.607977491Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5mXKhpLmaCwv2r1y08xfKZa","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.307745_92.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.4","timestamp":"2020-07-23T12:31:48.307745235Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yiZDGQu6X2F_Sk1M_XNX-aS","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.312319_95.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.4","timestamp":"2020-07-23T12:31:48.312319651Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yiZDGQu6X2F_Sk1M_XNX-aS","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.314524_96.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:48.314524642Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-Zgx0qRtoCp4uqoB0K4EqGk","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.318787_97.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:48.318787051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-Zgx0qRtoCp4uqoB0K4EqGk","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.342772_98.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.5","timestamp":"2020-07-23T12:31:48.342772112Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B4RZ8XZp_Vg_9H0AveP9SKb","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.347206_99.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.5","timestamp":"2020-07-23T12:31:48.347206077Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B4RZ8XZp_Vg_9H0AveP9SKb","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.349032_100.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:48.349032760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QSYVyw8rlDouP071b5aAg5c","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.352670_101.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:48.352670985Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QSYVyw8rlDouP071b5aAg5c","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.397942_102.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.6","timestamp":"2020-07-23T12:31:48.397942141Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Mxbry6a9nS2zmN7QABtJkqL","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.401908_103.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.6","timestamp":"2020-07-23T12:31:48.401908451Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Mxbry6a9nS2zmN7QABtJkqL","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507508.403730_104.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:48.403730015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nZJBxr1kdQBONurJVLbNJKj","ver":1},"status":{"status":"write"}}},{"id":"1_1595507508.407314_105.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:48.407314121Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nZJBxr1kdQBONurJVLbNJKj","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507515.909191_106.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.7","timestamp":"2020-07-23T12:31:55.909191875Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zoMNfl3WP6x69sG1A_a6fcE","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.913770_109.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.7","timestamp":"2020-07-23T12:31:55.913770150Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zoMNfl3WP6x69sG1A_a6fcE","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507515.915751_110.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:55.915751293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hU0LuGT_w40iAYldvPFrm0U","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.919252_111.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:55.919252707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hU0LuGT_w40iAYldvPFrm0U","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507515.950100_112.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.8","timestamp":"2020-07-23T12:31:55.950100507Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vltGzevaBLh8OZIhYyfm2CQ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.955276_113.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.8","timestamp":"2020-07-23T12:31:55.955276463Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vltGzevaBLh8OZIhYyfm2CQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507515.957617_114.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:55.957617299Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5r8WP0KmkR2EPhut7rsITD0","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.961660_115.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:55.961660024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5r8WP0KmkR2EPhut7rsITD0","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507515.989920_116.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.9","timestamp":"2020-07-23T12:31:55.989920273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zba4HrvvYXLNZLUZEEXyOjc","ver":1},"status":{"status":"write"}}},{"id":"1_1595507515.994394_117.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.9","timestamp":"2020-07-23T12:31:55.994394006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zba4HrvvYXLNZLUZEEXyOjc","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507515.996737_118.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:55.996737991Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XW9KXwz8j8Q2kDtlnNWtenR","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.002204_119.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.002204875Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XW9KXwz8j8Q2kDtlnNWtenR","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.038441_120.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.10","timestamp":"2020-07-23T12:31:56.038441936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yDod0geU58KV83pV2GT216-","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.042090_121.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.10","timestamp":"2020-07-23T12:31:56.042090624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yDod0geU58KV83pV2GT216-","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.044442_122.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:56.044442178Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SGxfEM8xG5yvSaheRQV2Ovd","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.047620_123.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:56.047620642Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SGxfEM8xG5yvSaheRQV2Ovd","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.098405_124.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.11","timestamp":"2020-07-23T12:31:56.098405415Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LUgRUmwvdm7gjw6CNXowxWA","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.102118_125.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.11","timestamp":"2020-07-23T12:31:56.102118377Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LUgRUmwvdm7gjw6CNXowxWA","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.103855_126.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:56.103855386Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_toR3dIX1U5fHjFxw5OD4lwp","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.106886_127.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:56.106886504Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_toR3dIX1U5fHjFxw5OD4lwp","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.198726_128.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.12","timestamp":"2020-07-23T12:31:56.198726163Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GxVDpKjDuPlGcusHQ2sPoDq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.202432_129.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.12","timestamp":"2020-07-23T12:31:56.202432137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GxVDpKjDuPlGcusHQ2sPoDq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.204287_130.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.204287823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wp9w1kaF3q9LFeeVvxwV4ZY","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.208680_131.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.208680506Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wp9w1kaF3q9LFeeVvxwV4ZY","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.310234_132.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.13","timestamp":"2020-07-23T12:31:56.310234859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pPnegWR9d5Ygh1JBGHNMLEp","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.314509_133.1","section":"bucket.instance","name":"pdivcf-4:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.13","timestamp":"2020-07-23T12:31:56.314509529Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pPnegWR9d5Ygh1JBGHNMLEp","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.316454_134.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:56.316454484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a0pbGO-wJ9KDNLVLy0l6mVw","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.321088_135.1","section":"bucket","name":"pdivcf-4","timestamp":"2020-07-23T12:31:56.321088724Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a0pbGO-wJ9KDNLVLy0l6mVw","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.376988_136.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.14","timestamp":"2020-07-23T12:31:56.376988037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5V1Wp4Ls0rjjCcGdYUoXxNK","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.381533_137.1","section":"bucket.instance","name":"pdivcf-5:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.14","timestamp":"2020-07-23T12:31:56.381533072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5V1Wp4Ls0rjjCcGdYUoXxNK","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.383610_138.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:56.383610350Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qC-7doEAfyxvcolW3WYIsNi","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.387474_139.1","section":"bucket","name":"pdivcf-5","timestamp":"2020-07-23T12:31:56.387474732Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qC-7doEAfyxvcolW3WYIsNi","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507516.482937_140.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.15","timestamp":"2020-07-23T12:31:56.482937943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6bW-esK6eOiMXZmLFiEt8dd","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.487703_141.1","section":"bucket.instance","name":"pdivcf-6:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.15","timestamp":"2020-07-23T12:31:56.487703511Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6bW-esK6eOiMXZmLFiEt8dd","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507516.489723_142.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.489723477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RXg1UMJCNsOx1Xd-Urfc0kP","ver":1},"status":{"status":"write"}}},{"id":"1_1595507516.493752_143.1","section":"bucket","name":"pdivcf-6","timestamp":"2020-07-23T12:31:56.493752921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RXg1UMJCNsOx1Xd-Urfc0kP","ver":1},"status":{"status":"abort"}}},{"id":"1_1595507519.200871_144.1","section":"bucket.instance","name":"pdivcf-7:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.16","timestamp":"2020-07-23T12:31:59.200871054Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5kZ1JmFLTWYDEE5vSb3YN9q","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.205867_147.1","section":"bucket.instance","name":"pdivcf-7:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.16","timestamp":"2020-07-23T12:31:59.205867247Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5kZ1JmFLTWYDEE5vSb3YN9q","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.208273_148.1","section":"bucket","name":"pdivcf-7","timestamp":"2020-07-23T12:31:59.208273943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.213060_149.1","section":"bucket","name":"pdivcf-7","timestamp":"2020-07-23T12:31:59.213060668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.236100_150.1","section":"bucket.instance","name":"pdivcf-8:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.17","timestamp":"2020-07-23T12:31:59.236100590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tCGnu2GnZ4klKs-yTkLNnbJ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.242796_151.1","section":"bucket.instance","name":"pdivcf-8:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.17","timestamp":"2020-07-23T12:31:59.242796192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tCGnu2GnZ4klKs-yTkLNnbJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.246769_152.1","section":"bucket","name":"pdivcf-8","timestamp":"2020-07-23T12:31:59.246769828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.253139_153.1","section":"bucket","name":"pdivcf-8","timestamp":"2020-07-23T12:31:59.253139879Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.306952_154.1","section":"bucket.instance","name":"pdivcf-9:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.18","timestamp":"2020-07-23T12:31:59.306952465Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kbWQDeLkSgvmeNGJdVE78Au","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.312907_155.1","section":"bucket.instance","name":"pdivcf-9:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.18","timestamp":"2020-07-23T12:31:59.312907044Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kbWQDeLkSgvmeNGJdVE78Au","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507519.315224_156.1","section":"bucket","name":"pdivcf-9","timestamp":"2020-07-23T12:31:59.315224409Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":1},"status":{"status":"write"}}},{"id":"1_1595507519.320666_157.1","section":"bucket","name":"pdivcf-9","timestamp":"2020-07-23T12:31:59.320666771Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507528.014327_158.1","section":"bucket","name":"pdivcf-7","timestamp":"2020-07-23T12:32:08.014327176Z","data":{"read_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":1},"write_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":2},"status":{"status":"remove"}}},{"id":"1_1595507528.020271_161.1","section":"bucket","name":"pdivcf-7","timestamp":"2020-07-23T12:32:08.020271414Z","data":{"read_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":1},"write_version":{"tag":"_puEurPihpFZM3LE8fEA9KAZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507528.077441_162.1","section":"bucket","name":"pdivcf-8","timestamp":"2020-07-23T12:32:08.077441314Z","data":{"read_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":1},"write_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":2},"status":{"status":"remove"}}},{"id":"1_1595507528.083398_163.1","section":"bucket","name":"pdivcf-8","timestamp":"2020-07-23T12:32:08.083398771Z","data":{"read_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":1},"write_version":{"tag":"_x2MZVS_zD-A6_Sw7NFWEpnO","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507528.169614_164.1","section":"bucket","name":"pdivcf-9","timestamp":"2020-07-23T12:32:08.169614550Z","data":{"read_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":1},"write_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":2},"status":{"status":"remove"}}},{"id":"1_1595507528.176421_165.1","section":"bucket","name":"pdivcf-9","timestamp":"2020-07-23T12:32:08.176421339Z","data":{"read_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":1},"write_version":{"tag":"_ELvfIpy-SQBEZDelbAK-7Uy","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507537.173491_46.1","section":"bucket.instance","name":"pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19","timestamp":"2020-07-23T12:32:17.173491511Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N7MWXq20eARG4bplabEWJWI","ver":1},"status":{"status":"write"}}},{"id":"1_1595507537.178224_55.1","section":"bucket.instance","name":"pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19","timestamp":"2020-07-23T12:32:17.178224061Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N7MWXq20eARG4bplabEWJWI","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507537.180408_56.1","section":"bucket","name":"pdivcf-10","timestamp":"2020-07-23T12:32:17.180408142Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eWs9kH85eL7ExiBJ3M7eDWt","ver":1},"status":{"status":"write"}}},{"id":"1_1595507537.185066_57.1","section":"bucket","name":"pdivcf-10","timestamp":"2020-07-23T12:32:17.185066160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eWs9kH85eL7ExiBJ3M7eDWt","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507538.116775_58.1","section":"bucket.instance","name":"pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20","timestamp":"2020-07-23T12:32:18.116775021Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0d0yAZmihVZhaW7NV6g_3kX","ver":1},"status":{"status":"write"}}},{"id":"1_1595507538.121315_59.1","section":"bucket.instance","name":"pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20","timestamp":"2020-07-23T12:32:18.121315724Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0d0yAZmihVZhaW7NV6g_3kX","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507538.123177_60.1","section":"bucket","name":"pdivcf-11","timestamp":"2020-07-23T12:32:18.123177285Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E1SDCqpJIo2DB_cVYeRt5Fj","ver":1},"status":{"status":"write"}}},{"id":"1_1595507538.130212_61.1","section":"bucket","name":"pdivcf-11","timestamp":"2020-07-23T12:32:18.130212887Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E1SDCqpJIo2DB_cVYeRt5Fj","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507538.842232_62.1","section":"bucket.instance","name":"pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21","timestamp":"2020-07-23T12:32:18.842232990Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DSr1_h9uAWM0Aupnq07MG-T","ver":1},"status":{"status":"write"}}},{"id":"1_1595507538.846555_63.1","section":"bucket.instance","name":"pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21","timestamp":"2020-07-23T12:32:18.846555026Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DSr1_h9uAWM0Aupnq07MG-T","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507538.848627_64.1","section":"bucket","name":"pdivcf-12","timestamp":"2020-07-23T12:32:18.848627622Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Idw7iMdTOFPnkuR0PvQ7oMk","ver":1},"status":{"status":"write"}}},{"id":"1_1595507538.853263_65.1","section":"bucket","name":"pdivcf-12","timestamp":"2020-07-23T12:32:18.853263990Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Idw7iMdTOFPnkuR0PvQ7oMk","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.772196_66.1","section":"bucket.instance","name":"pdivcf-13:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22","timestamp":"2020-07-23T12:32:37.772196573Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Na-elmOmjZsaFeM-ChUYR-E","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.776481_73.1","section":"bucket.instance","name":"pdivcf-13:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22","timestamp":"2020-07-23T12:32:37.776481061Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Na-elmOmjZsaFeM-ChUYR-E","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.778958_74.1","section":"bucket","name":"pdivcf-13","timestamp":"2020-07-23T12:32:37.778958954Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WAdUfnwKp-iQ-htLlTSCyaD","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.787127_75.1","section":"bucket","name":"pdivcf-13","timestamp":"2020-07-23T12:32:37.787127121Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WAdUfnwKp-iQ-htLlTSCyaD","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.815884_76.1","section":"bucket.instance","name":"pdivcf-14:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23","timestamp":"2020-07-23T12:32:37.815884493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BerFUu6TepMUW1OCfDzLt1L","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.820651_77.1","section":"bucket.instance","name":"pdivcf-14:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23","timestamp":"2020-07-23T12:32:37.820651075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BerFUu6TepMUW1OCfDzLt1L","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.823023_78.1","section":"bucket","name":"pdivcf-14","timestamp":"2020-07-23T12:32:37.823023440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBn5niBkGEfAMwLcYB5EkPd","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.828030_79.1","section":"bucket","name":"pdivcf-14","timestamp":"2020-07-23T12:32:37.828030042Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SBn5niBkGEfAMwLcYB5EkPd","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.877028_80.1","section":"bucket.instance","name":"pdivcf-15:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24","timestamp":"2020-07-23T12:32:37.877028266Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ph9awnaTrqKViZtfefg5dn8","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.881682_81.1","section":"bucket.instance","name":"pdivcf-15:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24","timestamp":"2020-07-23T12:32:37.881682539Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ph9awnaTrqKViZtfefg5dn8","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507557.884047_82.1","section":"bucket","name":"pdivcf-15","timestamp":"2020-07-23T12:32:37.884047505Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HaDbmpKCs0MvAJEp019L9md","ver":1},"status":{"status":"write"}}},{"id":"1_1595507557.889012_83.1","section":"bucket","name":"pdivcf-15","timestamp":"2020-07-23T12:32:37.889012624Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HaDbmpKCs0MvAJEp019L9md","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.476318_84.1","section":"bucket.instance","name":"pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25","timestamp":"2020-07-23T12:33:20.476318852Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.483791_98.1","section":"bucket.instance","name":"pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25","timestamp":"2020-07-23T12:33:20.483791941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.485654_99.1","section":"bucket","name":"pdivcf-16","timestamp":"2020-07-23T12:33:20.485654983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nNWx1-v33Cy8kx20mfEqCWM","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.490011_100.1","section":"bucket","name":"pdivcf-16","timestamp":"2020-07-23T12:33:20.490011274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nNWx1-v33Cy8kx20mfEqCWM","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.519518_101.1","section":"bucket.instance","name":"pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26","timestamp":"2020-07-23T12:33:20.519518015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.523662_102.1","section":"bucket.instance","name":"pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26","timestamp":"2020-07-23T12:33:20.523662515Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.526745_103.1","section":"bucket","name":"pdivcf-17","timestamp":"2020-07-23T12:33:20.526745182Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D9-gYM6k71SVeZ0yqDAlNyx","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.530907_104.1","section":"bucket","name":"pdivcf-17","timestamp":"2020-07-23T12:33:20.530907013Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D9-gYM6k71SVeZ0yqDAlNyx","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.570146_105.1","section":"bucket.instance","name":"pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27","timestamp":"2020-07-23T12:33:20.570146983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.575256_106.1","section":"bucket.instance","name":"pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27","timestamp":"2020-07-23T12:33:20.575256769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.577552_107.1","section":"bucket","name":"pdivcf-18","timestamp":"2020-07-23T12:33:20.577552876Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qfMtvuZVQROJ2UnUnTmsXWF","ver":1},"status":{"status":"write"}}},{"id":"1_1595507600.582467_108.1","section":"bucket","name":"pdivcf-18","timestamp":"2020-07-23T12:33:20.582467189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qfMtvuZVQROJ2UnUnTmsXWF","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507600.607611_109.1","section":"bucket.instance","name":"pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25","timestamp":"2020-07-23T12:33:20.607611890Z","data":{"read_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":1},"write_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":2},"status":{"status":"write"}}},{"id":"1_1595507600.614045_110.1","section":"bucket.instance","name":"pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25","timestamp":"2020-07-23T12:33:20.614045770Z","data":{"read_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":1},"write_version":{"tag":"_dpoYU47ApR8CSKXCGv7h-ad","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507600.619455_111.1","section":"bucket.instance","name":"pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26","timestamp":"2020-07-23T12:33:20.619455446Z","data":{"read_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":1},"write_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":2},"status":{"status":"write"}}},{"id":"1_1595507600.625710_112.1","section":"bucket.instance","name":"pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26","timestamp":"2020-07-23T12:33:20.625710444Z","data":{"read_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":1},"write_version":{"tag":"_6eCRjIgXspOmzAuW-3r9OZR","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507600.638571_113.1","section":"bucket.instance","name":"pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27","timestamp":"2020-07-23T12:33:20.638571023Z","data":{"read_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":1},"write_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":2},"status":{"status":"write"}}},{"id":"1_1595507600.647133_114.1","section":"bucket.instance","name":"pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27","timestamp":"2020-07-23T12:33:20.647133208Z","data":{"read_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":1},"write_version":{"tag":"_o7lBkZ70MIuG2t4k--7VNco","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507704.629540_179.1","section":"bucket.instance","name":"pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28","timestamp":"2020-07-23T12:35:04.629540333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":1},"status":{"status":"write"}}},{"id":"1_1595507704.634646_185.1","section":"bucket.instance","name":"pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28","timestamp":"2020-07-23T12:35:04.634646116Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507704.636917_186.1","section":"bucket","name":"pdivcf-19","timestamp":"2020-07-23T12:35:04.636917194Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5lux1Kcalu1aSjCSsZbbPBc","ver":1},"status":{"status":"write"}}},{"id":"1_1595507704.641982_187.1","section":"bucket","name":"pdivcf-19","timestamp":"2020-07-23T12:35:04.641982741Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5lux1Kcalu1aSjCSsZbbPBc","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507704.648928_188.1","section":"bucket.instance","name":"pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28","timestamp":"2020-07-23T12:35:04.648928543Z","data":{"read_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":1},"write_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":2},"status":{"status":"write"}}},{"id":"1_1595507704.656778_189.1","section":"bucket.instance","name":"pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28","timestamp":"2020-07-23T12:35:04.656778355Z","data":{"read_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":1},"write_version":{"tag":"_U0Qo2xqMTgl3e-3TTEpUtu9","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507770.568084_190.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:10.568084858Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":1},"status":{"status":"write"}}},{"id":"1_1595507770.572580_207.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:10.572580901Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507770.574547_208.1","section":"bucket","name":"pdivcf-20","timestamp":"2020-07-23T12:36:10.574547102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1GieWDzPbyhzYx6dzAbvM8Y","ver":1},"status":{"status":"write"}}},{"id":"1_1595507770.579148_209.1","section":"bucket","name":"pdivcf-20","timestamp":"2020-07-23T12:36:10.579148822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1GieWDzPbyhzYx6dzAbvM8Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507790.368503_210.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:30.368503148Z","data":{"read_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":1},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":2},"status":{"status":"write"}}},{"id":"1_1595507790.379387_217.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:30.379387206Z","data":{"read_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":1},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507811.952039_218.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:51.952039133Z","data":{"read_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":2},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":3},"status":{"status":"write"}}},{"id":"1_1595507811.963376_225.1","section":"bucket.instance","name":"pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29","timestamp":"2020-07-23T12:36:51.963376890Z","data":{"read_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":2},"write_version":{"tag":"_T6fP_s7x2erpKiPxnULpyXz","ver":3},"status":{"status":"complete"}}},{"id":"1_1595507833.977530_226.1","section":"bucket.instance","name":"pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30","timestamp":"2020-07-23T12:37:13.977530333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":1},"status":{"status":"write"}}},{"id":"1_1595507833.982409_233.1","section":"bucket.instance","name":"pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30","timestamp":"2020-07-23T12:37:13.982409662Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507833.984578_234.1","section":"bucket","name":"pdivcf-21","timestamp":"2020-07-23T12:37:13.984578572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxoN5QnXB8PzBQb6usiTCES","ver":1},"status":{"status":"write"}}},{"id":"1_1595507833.990265_235.1","section":"bucket","name":"pdivcf-21","timestamp":"2020-07-23T12:37:13.990265263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxoN5QnXB8PzBQb6usiTCES","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.016369_236.1","section":"bucket.instance","name":"pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31","timestamp":"2020-07-23T12:37:14.016369578Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507834.021064_237.1","section":"bucket.instance","name":"pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31","timestamp":"2020-07-23T12:37:14.021064652Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.023440_238.1","section":"bucket","name":"pdivcf-22","timestamp":"2020-07-23T12:37:14.023440018Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mOYhoW_mjO7mRP8PVQ6ZRH7","ver":1},"status":{"status":"write"}}},{"id":"1_1595507834.027916_239.1","section":"bucket","name":"pdivcf-22","timestamp":"2020-07-23T12:37:14.027916899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mOYhoW_mjO7mRP8PVQ6ZRH7","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.075022_240.1","section":"bucket.instance","name":"pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32","timestamp":"2020-07-23T12:37:14.075022229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":1},"status":{"status":"write"}}},{"id":"1_1595507834.080077_241.1","section":"bucket.instance","name":"pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32","timestamp":"2020-07-23T12:37:14.080077001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.082206_242.1","section":"bucket","name":"pdivcf-23","timestamp":"2020-07-23T12:37:14.082206069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C1euffFueI9tYW5vS7vMsgn","ver":1},"status":{"status":"write"}}},{"id":"1_1595507834.087443_243.1","section":"bucket","name":"pdivcf-23","timestamp":"2020-07-23T12:37:14.087443944Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C1euffFueI9tYW5vS7vMsgn","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507834.108675_244.1","section":"bucket.instance","name":"pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30","timestamp":"2020-07-23T12:37:14.108675749Z","data":{"read_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":1},"write_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":2},"status":{"status":"write"}}},{"id":"1_1595507834.113504_245.1","section":"bucket.instance","name":"pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30","timestamp":"2020-07-23T12:37:14.113504302Z","data":{"read_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":1},"write_version":{"tag":"_FfSUVqi-GRFU_BLUqDzGKlC","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507834.118829_246.1","section":"bucket.instance","name":"pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31","timestamp":"2020-07-23T12:37:14.118829894Z","data":{"read_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":1},"write_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":2},"status":{"status":"write"}}},{"id":"1_1595507834.124671_247.1","section":"bucket.instance","name":"pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31","timestamp":"2020-07-23T12:37:14.124671872Z","data":{"read_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":1},"write_version":{"tag":"_ilaYI1FiiTl74CmPd6mcbRZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507834.135668_248.1","section":"bucket.instance","name":"pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32","timestamp":"2020-07-23T12:37:14.135668915Z","data":{"read_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":1},"write_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":2},"status":{"status":"write"}}},{"id":"1_1595507834.142058_249.1","section":"bucket.instance","name":"pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32","timestamp":"2020-07-23T12:37:14.142058558Z","data":{"read_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":1},"write_version":{"tag":"_3_lQlM7n46yANqbF6c0iu23","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507862.301266_262.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.301266177Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.305752_275.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.305752811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.308557_276.1","section":"bucket","name":"pdivcf-24","timestamp":"2020-07-23T12:37:42.308557752Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9CmPcMhCq3DoG5OyzHLxsZp","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.313301_277.1","section":"bucket","name":"pdivcf-24","timestamp":"2020-07-23T12:37:42.313301958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9CmPcMhCq3DoG5OyzHLxsZp","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.341948_278.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.341948862Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.346312_279.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.346312178Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.348306_280.1","section":"bucket","name":"pdivcf-25","timestamp":"2020-07-23T12:37:42.348306812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_10rdqWmpSXM0gPEqJQLUt54","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.352897_281.1","section":"bucket","name":"pdivcf-25","timestamp":"2020-07-23T12:37:42.352897938Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_10rdqWmpSXM0gPEqJQLUt54","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.398608_282.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.398608049Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.402951_283.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.402951489Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.405025_284.1","section":"bucket","name":"pdivcf-26","timestamp":"2020-07-23T12:37:42.405025677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PtLBCckvoLRIiwjBffSRf4b","ver":1},"status":{"status":"write"}}},{"id":"1_1595507862.409024_285.1","section":"bucket","name":"pdivcf-26","timestamp":"2020-07-23T12:37:42.409024918Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PtLBCckvoLRIiwjBffSRf4b","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507862.436242_286.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.436242202Z","data":{"read_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":1},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":2},"status":{"status":"write"}}},{"id":"1_1595507862.443293_287.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.443293817Z","data":{"read_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":1},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507862.447803_288.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.447803976Z","data":{"read_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":2},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":3},"status":{"status":"write"}}},{"id":"1_1595507862.455267_289.1","section":"bucket.instance","name":"pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33","timestamp":"2020-07-23T12:37:42.455267808Z","data":{"read_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":2},"write_version":{"tag":"_vFyzdX7oDlx8uDmmzYUezxT","ver":3},"status":{"status":"complete"}}},{"id":"1_1595507862.461512_290.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.461512580Z","data":{"read_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":1},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":2},"status":{"status":"write"}}},{"id":"1_1595507862.469717_291.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.469717282Z","data":{"read_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":1},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507862.484236_292.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.484236274Z","data":{"read_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":2},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":3},"status":{"status":"write"}}},{"id":"1_1595507862.493648_293.1","section":"bucket.instance","name":"pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34","timestamp":"2020-07-23T12:37:42.493648043Z","data":{"read_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":2},"write_version":{"tag":"_9UeB5jKUR8uRZrL7kmlYbCL","ver":3},"status":{"status":"complete"}}},{"id":"1_1595507862.509518_294.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.509518426Z","data":{"read_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":1},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":2},"status":{"status":"write"}}},{"id":"1_1595507862.519332_295.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.519332057Z","data":{"read_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":1},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507862.547588_296.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.547588345Z","data":{"read_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":2},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":3},"status":{"status":"write"}}},{"id":"1_1595507862.561233_297.1","section":"bucket.instance","name":"pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35","timestamp":"2020-07-23T12:37:42.561233909Z","data":{"read_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":2},"write_version":{"tag":"_VGgKdThDc2jdr5TDn4SRY4t","ver":3},"status":{"status":"complete"}}},{"id":"1_1595507882.721482_301.1","section":"bucket.instance","name":"pdivcf-27:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.36","timestamp":"2020-07-23T12:38:02.721482894Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.726106_317.1","section":"bucket.instance","name":"pdivcf-27:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.36","timestamp":"2020-07-23T12:38:02.726106004Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.728482_318.1","section":"bucket","name":"pdivcf-27","timestamp":"2020-07-23T12:38:02.728482658Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0hGeTHEMYPqYlW8vmo8ZsB4","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.733340_319.1","section":"bucket","name":"pdivcf-27","timestamp":"2020-07-23T12:38:02.733340176Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0hGeTHEMYPqYlW8vmo8ZsB4","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.764598_320.1","section":"bucket.instance","name":"pdivcf-28:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37","timestamp":"2020-07-23T12:38:02.764598242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.768937_321.1","section":"bucket.instance","name":"pdivcf-28:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37","timestamp":"2020-07-23T12:38:02.768937393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.771175_322.1","section":"bucket","name":"pdivcf-28","timestamp":"2020-07-23T12:38:02.771175166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7Rwx9HRnK8Xo_lM2eZ_spRn","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.776360_323.1","section":"bucket","name":"pdivcf-28","timestamp":"2020-07-23T12:38:02.776360587Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7Rwx9HRnK8Xo_lM2eZ_spRn","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.827591_324.1","section":"bucket.instance","name":"pdivcf-29:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38","timestamp":"2020-07-23T12:38:02.827591764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.832410_325.1","section":"bucket.instance","name":"pdivcf-29:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38","timestamp":"2020-07-23T12:38:02.832410818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.834625_326.1","section":"bucket","name":"pdivcf-29","timestamp":"2020-07-23T12:38:02.834625226Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FBpeaUcoiAUeypLntXcZ6mt","ver":1},"status":{"status":"write"}}},{"id":"1_1595507882.839185_327.1","section":"bucket","name":"pdivcf-29","timestamp":"2020-07-23T12:38:02.839185239Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FBpeaUcoiAUeypLntXcZ6mt","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507882.863209_328.1","section":"bucket.instance","name":"pdivcf-27:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.36","timestamp":"2020-07-23T12:38:02.863209772Z","data":{"read_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":1},"write_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":2},"status":{"status":"write"}}},{"id":"1_1595507882.870400_329.1","section":"bucket.instance","name":"pdivcf-27:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.36","timestamp":"2020-07-23T12:38:02.870400054Z","data":{"read_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":1},"write_version":{"tag":"_GcO6mlyijVf8WprEal5O4d9","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507882.925804_330.1","section":"bucket.instance","name":"pdivcf-28:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37","timestamp":"2020-07-23T12:38:02.925804207Z","data":{"read_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":1},"write_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":2},"status":{"status":"write"}}},{"id":"1_1595507882.936647_331.1","section":"bucket.instance","name":"pdivcf-28:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37","timestamp":"2020-07-23T12:38:02.936647924Z","data":{"read_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":1},"write_version":{"tag":"_OGCsv5InUkCDSd6U1IaMguq","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507882.996474_332.1","section":"bucket.instance","name":"pdivcf-29:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38","timestamp":"2020-07-23T12:38:02.996474008Z","data":{"read_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":1},"write_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.007982_333.1","section":"bucket.instance","name":"pdivcf-29:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38","timestamp":"2020-07-23T12:38:03.007982412Z","data":{"read_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":1},"write_version":{"tag":"_Iv-aVuaylDjLxNVgxFRe6XY","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507883.085277_334.1","section":"bucket.instance","name":"pdivcf-30:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.39","timestamp":"2020-07-23T12:38:03.085277631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.094971_335.1","section":"bucket.instance","name":"pdivcf-30:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.39","timestamp":"2020-07-23T12:38:03.094971334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.097157_336.1","section":"bucket","name":"pdivcf-30","timestamp":"2020-07-23T12:38:03.097157211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pkn26B7VC8HessJ3RMQLyfb","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.101146_337.1","section":"bucket","name":"pdivcf-30","timestamp":"2020-07-23T12:38:03.101146718Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pkn26B7VC8HessJ3RMQLyfb","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.123841_338.1","section":"bucket.instance","name":"pdivcf-31:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40","timestamp":"2020-07-23T12:38:03.123841176Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.128855_339.1","section":"bucket.instance","name":"pdivcf-31:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40","timestamp":"2020-07-23T12:38:03.128855661Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.131009_340.1","section":"bucket","name":"pdivcf-31","timestamp":"2020-07-23T12:38:03.131009569Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JpktLwZXqe4IoNIvo0sPRIm","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.135379_341.1","section":"bucket","name":"pdivcf-31","timestamp":"2020-07-23T12:38:03.135379648Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JpktLwZXqe4IoNIvo0sPRIm","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.179493_342.1","section":"bucket.instance","name":"pdivcf-32:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41","timestamp":"2020-07-23T12:38:03.179493154Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.183875_343.1","section":"bucket.instance","name":"pdivcf-32:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41","timestamp":"2020-07-23T12:38:03.183875771Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.186063_344.1","section":"bucket","name":"pdivcf-32","timestamp":"2020-07-23T12:38:03.186063685Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2yNrr1LVREKqlHu0zys3qhJ","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.190110_345.1","section":"bucket","name":"pdivcf-32","timestamp":"2020-07-23T12:38:03.190110469Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2yNrr1LVREKqlHu0zys3qhJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.215750_346.1","section":"bucket.instance","name":"pdivcf-30:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.39","timestamp":"2020-07-23T12:38:03.215750124Z","data":{"read_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":1},"write_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.221277_347.1","section":"bucket.instance","name":"pdivcf-30:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.39","timestamp":"2020-07-23T12:38:03.221277172Z","data":{"read_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":1},"write_version":{"tag":"_cFMZKyzxStTSjJ0sKYfxVra","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507883.274318_348.1","section":"bucket.instance","name":"pdivcf-31:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40","timestamp":"2020-07-23T12:38:03.274318325Z","data":{"read_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":1},"write_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.286399_349.1","section":"bucket.instance","name":"pdivcf-31:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40","timestamp":"2020-07-23T12:38:03.286399254Z","data":{"read_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":1},"write_version":{"tag":"_74E2-yo8mNDXOpuYXpvfwgG","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507883.346515_350.1","section":"bucket.instance","name":"pdivcf-32:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41","timestamp":"2020-07-23T12:38:03.346515051Z","data":{"read_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":1},"write_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.356462_351.1","section":"bucket.instance","name":"pdivcf-32:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41","timestamp":"2020-07-23T12:38:03.356462902Z","data":{"read_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":1},"write_version":{"tag":"_ZuDEOa4qhSsWMnKISl72FU3","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507883.435617_352.1","section":"bucket.instance","name":"pdivcf-33:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.42","timestamp":"2020-07-23T12:38:03.435617749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.449308_353.1","section":"bucket.instance","name":"pdivcf-33:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.42","timestamp":"2020-07-23T12:38:03.449308210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.462520_354.1","section":"bucket","name":"pdivcf-33","timestamp":"2020-07-23T12:38:03.462520643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TJwU15-56F6OaBLbx9XF7mq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.466784_355.1","section":"bucket","name":"pdivcf-33","timestamp":"2020-07-23T12:38:03.466784917Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TJwU15-56F6OaBLbx9XF7mq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.564568_356.1","section":"bucket.instance","name":"pdivcf-34:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43","timestamp":"2020-07-23T12:38:03.564568818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.569408_357.1","section":"bucket.instance","name":"pdivcf-34:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43","timestamp":"2020-07-23T12:38:03.569408997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.571379_358.1","section":"bucket","name":"pdivcf-34","timestamp":"2020-07-23T12:38:03.571379967Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QaQnidDCu-eD3G2dQaeWRxh","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.575816_359.1","section":"bucket","name":"pdivcf-34","timestamp":"2020-07-23T12:38:03.575816752Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QaQnidDCu-eD3G2dQaeWRxh","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.617917_360.1","section":"bucket.instance","name":"pdivcf-35:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44","timestamp":"2020-07-23T12:38:03.617917681Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.622832_361.1","section":"bucket.instance","name":"pdivcf-35:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44","timestamp":"2020-07-23T12:38:03.622832163Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.625040_362.1","section":"bucket","name":"pdivcf-35","timestamp":"2020-07-23T12:38:03.625040094Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f8RF8Y341PAQo2j4DYcRQfj","ver":1},"status":{"status":"write"}}},{"id":"1_1595507883.630457_363.1","section":"bucket","name":"pdivcf-35","timestamp":"2020-07-23T12:38:03.630457468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f8RF8Y341PAQo2j4DYcRQfj","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507883.656588_364.1","section":"bucket.instance","name":"pdivcf-33:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.42","timestamp":"2020-07-23T12:38:03.656588365Z","data":{"read_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":1},"write_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":2},"status":{"status":"write"}}},{"id":"1_1595507883.663207_365.1","section":"bucket.instance","name":"pdivcf-33:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.42","timestamp":"2020-07-23T12:38:03.663207212Z","data":{"read_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":1},"write_version":{"tag":"_uK5gmHo5eEldoEO_I-CWSJV","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507884.713429_366.1","section":"bucket.instance","name":"pdivcf-34:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43","timestamp":"2020-07-23T12:38:04.713429561Z","data":{"read_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":1},"write_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":2},"status":{"status":"write"}}},{"id":"1_1595507884.725262_367.1","section":"bucket.instance","name":"pdivcf-34:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43","timestamp":"2020-07-23T12:38:04.725262276Z","data":{"read_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":1},"write_version":{"tag":"_Y2HWaqARl3awMVvdt2VsaNr","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507885.864935_368.1","section":"bucket.instance","name":"pdivcf-35:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44","timestamp":"2020-07-23T12:38:05.864935216Z","data":{"read_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":1},"write_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":2},"status":{"status":"write"}}},{"id":"1_1595507885.876462_369.1","section":"bucket.instance","name":"pdivcf-35:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44","timestamp":"2020-07-23T12:38:05.876462797Z","data":{"read_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":1},"write_version":{"tag":"_VuQBKPactetMJJmeuORmZsf","ver":2},"status":{"status":"complete"}}},{"id":"1_1595507886.300008_370.1","section":"bucket.instance","name":"pdivcf-36:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45","timestamp":"2020-07-23T12:38:06.300008664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ps7FDMLj9P7FIJ3TQF3GFZk","ver":1},"status":{"status":"write"}}},{"id":"1_1595507886.304501_371.1","section":"bucket.instance","name":"pdivcf-36:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45","timestamp":"2020-07-23T12:38:06.304501769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ps7FDMLj9P7FIJ3TQF3GFZk","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507886.306748_372.1","section":"bucket","name":"pdivcf-36","timestamp":"2020-07-23T12:38:06.306748955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LrkNjaTY34zXjmNUserxO9w","ver":1},"status":{"status":"write"}}},{"id":"1_1595507886.311229_373.1","section":"bucket","name":"pdivcf-36","timestamp":"2020-07-23T12:38:06.311229165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LrkNjaTY34zXjmNUserxO9w","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507887.131065_374.1","section":"bucket.instance","name":"pdivcf-37:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46","timestamp":"2020-07-23T12:38:07.131065889Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Voz6xrKT44N52QQm_1jkHXq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507887.136159_375.1","section":"bucket.instance","name":"pdivcf-37:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46","timestamp":"2020-07-23T12:38:07.136159187Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Voz6xrKT44N52QQm_1jkHXq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507887.138199_376.1","section":"bucket","name":"pdivcf-37","timestamp":"2020-07-23T12:38:07.138199539Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GgAC1P5cFFfaEIRduZflSLw","ver":1},"status":{"status":"write"}}},{"id":"1_1595507887.142636_377.1","section":"bucket","name":"pdivcf-37","timestamp":"2020-07-23T12:38:07.142636002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GgAC1P5cFFfaEIRduZflSLw","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507887.514178_378.1","section":"bucket.instance","name":"pdivcf-38:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47","timestamp":"2020-07-23T12:38:07.514178897Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oY59xvqz8ppnd0uahVk4x_2","ver":1},"status":{"status":"write"}}},{"id":"1_1595507887.518533_379.1","section":"bucket.instance","name":"pdivcf-38:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47","timestamp":"2020-07-23T12:38:07.518533828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oY59xvqz8ppnd0uahVk4x_2","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507887.520752_380.1","section":"bucket","name":"pdivcf-38","timestamp":"2020-07-23T12:38:07.520752755Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N85MuH5ymUQx_bvR1jEV5oS","ver":1},"status":{"status":"write"}}},{"id":"1_1595507887.526314_381.1","section":"bucket","name":"pdivcf-38","timestamp":"2020-07-23T12:38:07.526314606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N85MuH5ymUQx_bvR1jEV5oS","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.085257_382.1","section":"bucket.instance","name":"pdivcf-39:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.48","timestamp":"2020-07-23T12:38:10.085257016Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_47jziFa-v3TzgKog19keFK3","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.090510_384.1","section":"bucket.instance","name":"pdivcf-39:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.48","timestamp":"2020-07-23T12:38:10.090510647Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_47jziFa-v3TzgKog19keFK3","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.092807_385.1","section":"bucket","name":"pdivcf-39","timestamp":"2020-07-23T12:38:10.092807212Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J6h6UEs-Kdv67K6lZ-DDqe_","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.098065_386.1","section":"bucket","name":"pdivcf-39","timestamp":"2020-07-23T12:38:10.098065606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J6h6UEs-Kdv67K6lZ-DDqe_","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.130438_387.1","section":"bucket.instance","name":"pdivcf-40:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.49","timestamp":"2020-07-23T12:38:10.130438151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jBkqIWgSeJygUbxw4jdVMVd","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.135039_388.1","section":"bucket.instance","name":"pdivcf-40:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.49","timestamp":"2020-07-23T12:38:10.135039327Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jBkqIWgSeJygUbxw4jdVMVd","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.137937_389.1","section":"bucket","name":"pdivcf-40","timestamp":"2020-07-23T12:38:10.137937297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Nsh2aQliOMNd_qa-DYEFjIr","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.146156_390.1","section":"bucket","name":"pdivcf-40","timestamp":"2020-07-23T12:38:10.146156264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Nsh2aQliOMNd_qa-DYEFjIr","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.204712_391.1","section":"bucket.instance","name":"pdivcf-41:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.50","timestamp":"2020-07-23T12:38:10.204712051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jjdhZX1cAX8QT7odbfHWP9O","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.209370_392.1","section":"bucket.instance","name":"pdivcf-41:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.50","timestamp":"2020-07-23T12:38:10.209370112Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jjdhZX1cAX8QT7odbfHWP9O","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507890.211763_393.1","section":"bucket","name":"pdivcf-41","timestamp":"2020-07-23T12:38:10.211763358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Voj1i0FYSZ0L7r0fMh4Tdjk","ver":1},"status":{"status":"write"}}},{"id":"1_1595507890.217061_394.1","section":"bucket","name":"pdivcf-41","timestamp":"2020-07-23T12:38:10.217061137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Voj1i0FYSZ0L7r0fMh4Tdjk","ver":1},"status":{"status":"complete"}}}] 2020-07-23T12:40:40.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '30fc0b4a-5063-4d9f-a69b-5de0a471142a', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:40.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '30fc0b4a-5063-4d9f-a69b-5de0a471142a', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:40.471 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 30fc0b4a-5063-4d9f-a69b-5de0a471142a --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:40.511 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:40.511 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:40.784 INFO:teuthology.orchestra.run.smithi137.stdout:[{"id":"1_1595507963.344964_258.1","section":"bucket.instance","name":"pdivcf-42:42997773-3698-4448-bf7b-a00a5846d584.6302.1","timestamp":"2020-07-23T12:39:23.344964051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jP5Jrb2o1AWvbvoEzZp-_vD","ver":1},"status":{"status":"write"}}},{"id":"1_1595507963.349367_259.1","section":"bucket.instance","name":"pdivcf-42:42997773-3698-4448-bf7b-a00a5846d584.6302.1","timestamp":"2020-07-23T12:39:23.349367752Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jP5Jrb2o1AWvbvoEzZp-_vD","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507963.351468_260.1","section":"bucket","name":"pdivcf-42","timestamp":"2020-07-23T12:39:23.351468911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MdTgoLzEpc-9UnQFBhQrXOq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507963.355723_261.1","section":"bucket","name":"pdivcf-42","timestamp":"2020-07-23T12:39:23.355723211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MdTgoLzEpc-9UnQFBhQrXOq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507963.407747_262.1","section":"bucket.instance","name":"pdivcf-43:42997773-3698-4448-bf7b-a00a5846d584.6302.2","timestamp":"2020-07-23T12:39:23.407747257Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f9RC4jFVqAMfnvZZDpzVuhb","ver":1},"status":{"status":"write"}}},{"id":"1_1595507963.412680_263.1","section":"bucket.instance","name":"pdivcf-43:42997773-3698-4448-bf7b-a00a5846d584.6302.2","timestamp":"2020-07-23T12:39:23.412680427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f9RC4jFVqAMfnvZZDpzVuhb","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507963.415163_264.1","section":"bucket","name":"pdivcf-43","timestamp":"2020-07-23T12:39:23.415163333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lJDe9sZI_owqmlIXltmpY1G","ver":1},"status":{"status":"write"}}},{"id":"1_1595507963.420183_265.1","section":"bucket","name":"pdivcf-43","timestamp":"2020-07-23T12:39:23.420183863Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lJDe9sZI_owqmlIXltmpY1G","ver":1},"status":{"status":"complete"}}}] 2020-07-23T12:40:40.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '30fc0b4a-5063-4d9f-a69b-5de0a471142a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:40.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '30fc0b4a-5063-4d9f-a69b-5de0a471142a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:40.787 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 30fc0b4a-5063-4d9f-a69b-5de0a471142a --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:40.832 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:40.832 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:41.086 INFO:teuthology.orchestra.run.smithi133.stdout:[{"id":"1_1595507963.344964_258.1","section":"bucket.instance","name":"pdivcf-42:42997773-3698-4448-bf7b-a00a5846d584.6302.1","timestamp":"2020-07-23T12:39:23.344964051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jP5Jrb2o1AWvbvoEzZp-_vD","ver":1},"status":{"status":"write"}}},{"id":"1_1595507963.349367_259.1","section":"bucket.instance","name":"pdivcf-42:42997773-3698-4448-bf7b-a00a5846d584.6302.1","timestamp":"2020-07-23T12:39:23.349367752Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jP5Jrb2o1AWvbvoEzZp-_vD","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507963.351468_260.1","section":"bucket","name":"pdivcf-42","timestamp":"2020-07-23T12:39:23.351468911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MdTgoLzEpc-9UnQFBhQrXOq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507963.355723_261.1","section":"bucket","name":"pdivcf-42","timestamp":"2020-07-23T12:39:23.355723211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MdTgoLzEpc-9UnQFBhQrXOq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507963.407747_262.1","section":"bucket.instance","name":"pdivcf-43:42997773-3698-4448-bf7b-a00a5846d584.6302.2","timestamp":"2020-07-23T12:39:23.407747257Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f9RC4jFVqAMfnvZZDpzVuhb","ver":1},"status":{"status":"write"}}},{"id":"1_1595507963.412680_263.1","section":"bucket.instance","name":"pdivcf-43:42997773-3698-4448-bf7b-a00a5846d584.6302.2","timestamp":"2020-07-23T12:39:23.412680427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f9RC4jFVqAMfnvZZDpzVuhb","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507963.415163_264.1","section":"bucket","name":"pdivcf-43","timestamp":"2020-07-23T12:39:23.415163333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lJDe9sZI_owqmlIXltmpY1G","ver":1},"status":{"status":"write"}}},{"id":"1_1595507963.420183_265.1","section":"bucket","name":"pdivcf-43","timestamp":"2020-07-23T12:39:23.420183863Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lJDe9sZI_owqmlIXltmpY1G","ver":1},"status":{"status":"complete"}}}] 2020-07-23T12:40:41.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '30fc0b4a-5063-4d9f-a69b-5de0a471142a', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:41.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '30fc0b4a-5063-4d9f-a69b-5de0a471142a', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:41.088 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 30fc0b4a-5063-4d9f-a69b-5de0a471142a --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:41.127 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:41.128 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:41.396 INFO:teuthology.orchestra.run.smithi137.stdout:[{"id":"1_1595507963.344964_258.1","section":"bucket.instance","name":"pdivcf-42:42997773-3698-4448-bf7b-a00a5846d584.6302.1","timestamp":"2020-07-23T12:39:23.344964051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jP5Jrb2o1AWvbvoEzZp-_vD","ver":1},"status":{"status":"write"}}},{"id":"1_1595507963.349367_259.1","section":"bucket.instance","name":"pdivcf-42:42997773-3698-4448-bf7b-a00a5846d584.6302.1","timestamp":"2020-07-23T12:39:23.349367752Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jP5Jrb2o1AWvbvoEzZp-_vD","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507963.351468_260.1","section":"bucket","name":"pdivcf-42","timestamp":"2020-07-23T12:39:23.351468911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MdTgoLzEpc-9UnQFBhQrXOq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507963.355723_261.1","section":"bucket","name":"pdivcf-42","timestamp":"2020-07-23T12:39:23.355723211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MdTgoLzEpc-9UnQFBhQrXOq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507963.407747_262.1","section":"bucket.instance","name":"pdivcf-43:42997773-3698-4448-bf7b-a00a5846d584.6302.2","timestamp":"2020-07-23T12:39:23.407747257Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f9RC4jFVqAMfnvZZDpzVuhb","ver":1},"status":{"status":"write"}}},{"id":"1_1595507963.412680_263.1","section":"bucket.instance","name":"pdivcf-43:42997773-3698-4448-bf7b-a00a5846d584.6302.2","timestamp":"2020-07-23T12:39:23.412680427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f9RC4jFVqAMfnvZZDpzVuhb","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507963.415163_264.1","section":"bucket","name":"pdivcf-43","timestamp":"2020-07-23T12:39:23.415163333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lJDe9sZI_owqmlIXltmpY1G","ver":1},"status":{"status":"write"}}},{"id":"1_1595507963.420183_265.1","section":"bucket","name":"pdivcf-43","timestamp":"2020-07-23T12:39:23.420183863Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lJDe9sZI_owqmlIXltmpY1G","ver":1},"status":{"status":"complete"}}}] 2020-07-23T12:40:41.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '30fc0b4a-5063-4d9f-a69b-5de0a471142a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:41.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '30fc0b4a-5063-4d9f-a69b-5de0a471142a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:41.399 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 30fc0b4a-5063-4d9f-a69b-5de0a471142a --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:41.439 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:41.440 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:41.682 INFO:teuthology.orchestra.run.smithi133.stdout:[{"id":"1_1595507963.344964_258.1","section":"bucket.instance","name":"pdivcf-42:42997773-3698-4448-bf7b-a00a5846d584.6302.1","timestamp":"2020-07-23T12:39:23.344964051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jP5Jrb2o1AWvbvoEzZp-_vD","ver":1},"status":{"status":"write"}}},{"id":"1_1595507963.349367_259.1","section":"bucket.instance","name":"pdivcf-42:42997773-3698-4448-bf7b-a00a5846d584.6302.1","timestamp":"2020-07-23T12:39:23.349367752Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jP5Jrb2o1AWvbvoEzZp-_vD","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507963.351468_260.1","section":"bucket","name":"pdivcf-42","timestamp":"2020-07-23T12:39:23.351468911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MdTgoLzEpc-9UnQFBhQrXOq","ver":1},"status":{"status":"write"}}},{"id":"1_1595507963.355723_261.1","section":"bucket","name":"pdivcf-42","timestamp":"2020-07-23T12:39:23.355723211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MdTgoLzEpc-9UnQFBhQrXOq","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507963.407747_262.1","section":"bucket.instance","name":"pdivcf-43:42997773-3698-4448-bf7b-a00a5846d584.6302.2","timestamp":"2020-07-23T12:39:23.407747257Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f9RC4jFVqAMfnvZZDpzVuhb","ver":1},"status":{"status":"write"}}},{"id":"1_1595507963.412680_263.1","section":"bucket.instance","name":"pdivcf-43:42997773-3698-4448-bf7b-a00a5846d584.6302.2","timestamp":"2020-07-23T12:39:23.412680427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f9RC4jFVqAMfnvZZDpzVuhb","ver":1},"status":{"status":"complete"}}},{"id":"1_1595507963.415163_264.1","section":"bucket","name":"pdivcf-43","timestamp":"2020-07-23T12:39:23.415163333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lJDe9sZI_owqmlIXltmpY1G","ver":1},"status":{"status":"write"}}},{"id":"1_1595507963.420183_265.1","section":"bucket","name":"pdivcf-43","timestamp":"2020-07-23T12:39:23.420183863Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lJDe9sZI_owqmlIXltmpY1G","ver":1},"status":{"status":"complete"}}}] 2020-07-23T12:40:41.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'a2b03b35-661e-4182-91b6-d60b3b65f694', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:41.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'a2b03b35-661e-4182-91b6-d60b3b65f694', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:41.685 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period a2b03b35-661e-4182-91b6-d60b3b65f694 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:41.724 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:41.724 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:41.982 INFO:teuthology.orchestra.run.smithi137.stdout:[{"id":"1_1595508029.904847_101.1","section":"bucket.instance","name":"pdivcf-44:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.1","timestamp":"2020-07-23T12:40:29.904847523Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TpwQ1tJ2cig1TljtWPe-ZJ2","ver":1},"status":{"status":"write"}}},{"id":"1_1595508029.909716_102.1","section":"bucket.instance","name":"pdivcf-44:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.1","timestamp":"2020-07-23T12:40:29.909716552Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TpwQ1tJ2cig1TljtWPe-ZJ2","ver":1},"status":{"status":"complete"}}},{"id":"1_1595508029.914676_103.1","section":"bucket","name":"pdivcf-44","timestamp":"2020-07-23T12:40:29.914676300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w3Xle4TcqjI9Z_tbmt5PWXo","ver":1},"status":{"status":"write"}}},{"id":"1_1595508029.919844_104.1","section":"bucket","name":"pdivcf-44","timestamp":"2020-07-23T12:40:29.919844301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w3Xle4TcqjI9Z_tbmt5PWXo","ver":1},"status":{"status":"complete"}}},{"id":"1_1595508029.970016_105.1","section":"bucket.instance","name":"pdivcf-45:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.2","timestamp":"2020-07-23T12:40:29.970016048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_d1vci0GScMp6NjcyM03X3q4","ver":1},"status":{"status":"write"}}},{"id":"1_1595508029.974264_106.1","section":"bucket.instance","name":"pdivcf-45:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.2","timestamp":"2020-07-23T12:40:29.974264736Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_d1vci0GScMp6NjcyM03X3q4","ver":1},"status":{"status":"complete"}}},{"id":"1_1595508029.976208_107.1","section":"bucket","name":"pdivcf-45","timestamp":"2020-07-23T12:40:29.976208951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_siSMT7ej6dpnt6INyalwI6i","ver":1},"status":{"status":"write"}}},{"id":"1_1595508029.980256_108.1","section":"bucket","name":"pdivcf-45","timestamp":"2020-07-23T12:40:29.980256633Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_siSMT7ej6dpnt6INyalwI6i","ver":1},"status":{"status":"complete"}}}] 2020-07-23T12:40:41.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'a2b03b35-661e-4182-91b6-d60b3b65f694', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:41.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'a2b03b35-661e-4182-91b6-d60b3b65f694', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:41.985 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period a2b03b35-661e-4182-91b6-d60b3b65f694 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:42.024 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:42.024 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:42.282 INFO:teuthology.orchestra.run.smithi133.stdout:[{"id":"1_1595508029.904847_101.1","section":"bucket.instance","name":"pdivcf-44:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.1","timestamp":"2020-07-23T12:40:29.904847523Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TpwQ1tJ2cig1TljtWPe-ZJ2","ver":1},"status":{"status":"write"}}},{"id":"1_1595508029.909716_102.1","section":"bucket.instance","name":"pdivcf-44:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.1","timestamp":"2020-07-23T12:40:29.909716552Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TpwQ1tJ2cig1TljtWPe-ZJ2","ver":1},"status":{"status":"complete"}}},{"id":"1_1595508029.914676_103.1","section":"bucket","name":"pdivcf-44","timestamp":"2020-07-23T12:40:29.914676300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w3Xle4TcqjI9Z_tbmt5PWXo","ver":1},"status":{"status":"write"}}},{"id":"1_1595508029.919844_104.1","section":"bucket","name":"pdivcf-44","timestamp":"2020-07-23T12:40:29.919844301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w3Xle4TcqjI9Z_tbmt5PWXo","ver":1},"status":{"status":"complete"}}},{"id":"1_1595508029.970016_105.1","section":"bucket.instance","name":"pdivcf-45:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.2","timestamp":"2020-07-23T12:40:29.970016048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_d1vci0GScMp6NjcyM03X3q4","ver":1},"status":{"status":"write"}}},{"id":"1_1595508029.974264_106.1","section":"bucket.instance","name":"pdivcf-45:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.2","timestamp":"2020-07-23T12:40:29.974264736Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_d1vci0GScMp6NjcyM03X3q4","ver":1},"status":{"status":"complete"}}},{"id":"1_1595508029.976208_107.1","section":"bucket","name":"pdivcf-45","timestamp":"2020-07-23T12:40:29.976208951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_siSMT7ej6dpnt6INyalwI6i","ver":1},"status":{"status":"write"}}},{"id":"1_1595508029.980256_108.1","section":"bucket","name":"pdivcf-45","timestamp":"2020-07-23T12:40:29.980256633Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_siSMT7ej6dpnt6INyalwI6i","ver":1},"status":{"status":"complete"}}}] 2020-07-23T12:40:42.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'a2b03b35-661e-4182-91b6-d60b3b65f694', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:42.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'a2b03b35-661e-4182-91b6-d60b3b65f694', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:42.284 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period a2b03b35-661e-4182-91b6-d60b3b65f694 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:42.321 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:42.322 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:42.599 INFO:teuthology.orchestra.run.smithi137.stdout:[{"id":"1_1595508029.904847_101.1","section":"bucket.instance","name":"pdivcf-44:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.1","timestamp":"2020-07-23T12:40:29.904847523Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TpwQ1tJ2cig1TljtWPe-ZJ2","ver":1},"status":{"status":"write"}}},{"id":"1_1595508029.909716_102.1","section":"bucket.instance","name":"pdivcf-44:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.1","timestamp":"2020-07-23T12:40:29.909716552Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TpwQ1tJ2cig1TljtWPe-ZJ2","ver":1},"status":{"status":"complete"}}},{"id":"1_1595508029.914676_103.1","section":"bucket","name":"pdivcf-44","timestamp":"2020-07-23T12:40:29.914676300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w3Xle4TcqjI9Z_tbmt5PWXo","ver":1},"status":{"status":"write"}}},{"id":"1_1595508029.919844_104.1","section":"bucket","name":"pdivcf-44","timestamp":"2020-07-23T12:40:29.919844301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w3Xle4TcqjI9Z_tbmt5PWXo","ver":1},"status":{"status":"complete"}}},{"id":"1_1595508029.970016_105.1","section":"bucket.instance","name":"pdivcf-45:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.2","timestamp":"2020-07-23T12:40:29.970016048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_d1vci0GScMp6NjcyM03X3q4","ver":1},"status":{"status":"write"}}},{"id":"1_1595508029.974264_106.1","section":"bucket.instance","name":"pdivcf-45:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.2","timestamp":"2020-07-23T12:40:29.974264736Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_d1vci0GScMp6NjcyM03X3q4","ver":1},"status":{"status":"complete"}}},{"id":"1_1595508029.976208_107.1","section":"bucket","name":"pdivcf-45","timestamp":"2020-07-23T12:40:29.976208951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_siSMT7ej6dpnt6INyalwI6i","ver":1},"status":{"status":"write"}}},{"id":"1_1595508029.980256_108.1","section":"bucket","name":"pdivcf-45","timestamp":"2020-07-23T12:40:29.980256633Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_siSMT7ej6dpnt6INyalwI6i","ver":1},"status":{"status":"complete"}}}] 2020-07-23T12:40:42.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'a2b03b35-661e-4182-91b6-d60b3b65f694', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:42.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'a2b03b35-661e-4182-91b6-d60b3b65f694', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:42.601 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period a2b03b35-661e-4182-91b6-d60b3b65f694 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:42.641 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:42.642 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:42.900 INFO:teuthology.orchestra.run.smithi133.stdout:[{"id":"1_1595508029.904847_101.1","section":"bucket.instance","name":"pdivcf-44:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.1","timestamp":"2020-07-23T12:40:29.904847523Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TpwQ1tJ2cig1TljtWPe-ZJ2","ver":1},"status":{"status":"write"}}},{"id":"1_1595508029.909716_102.1","section":"bucket.instance","name":"pdivcf-44:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.1","timestamp":"2020-07-23T12:40:29.909716552Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TpwQ1tJ2cig1TljtWPe-ZJ2","ver":1},"status":{"status":"complete"}}},{"id":"1_1595508029.914676_103.1","section":"bucket","name":"pdivcf-44","timestamp":"2020-07-23T12:40:29.914676300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w3Xle4TcqjI9Z_tbmt5PWXo","ver":1},"status":{"status":"write"}}},{"id":"1_1595508029.919844_104.1","section":"bucket","name":"pdivcf-44","timestamp":"2020-07-23T12:40:29.919844301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_w3Xle4TcqjI9Z_tbmt5PWXo","ver":1},"status":{"status":"complete"}}},{"id":"1_1595508029.970016_105.1","section":"bucket.instance","name":"pdivcf-45:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.2","timestamp":"2020-07-23T12:40:29.970016048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_d1vci0GScMp6NjcyM03X3q4","ver":1},"status":{"status":"write"}}},{"id":"1_1595508029.974264_106.1","section":"bucket.instance","name":"pdivcf-45:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.2","timestamp":"2020-07-23T12:40:29.974264736Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_d1vci0GScMp6NjcyM03X3q4","ver":1},"status":{"status":"complete"}}},{"id":"1_1595508029.976208_107.1","section":"bucket","name":"pdivcf-45","timestamp":"2020-07-23T12:40:29.976208951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_siSMT7ej6dpnt6INyalwI6i","ver":1},"status":{"status":"write"}}},{"id":"1_1595508029.980256_108.1","section":"bucket","name":"pdivcf-45","timestamp":"2020-07-23T12:40:29.980256633Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_siSMT7ej6dpnt6INyalwI6i","ver":1},"status":{"status":"complete"}}}] 2020-07-23T12:40:42.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:40:42.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:40:42.902 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:40:42.939 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:42.939 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:43.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:40:43.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:40:43.235 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:40:43.270 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:43.271 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:43.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:40:43.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:40:43.651 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:40:43.689 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:43.689 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:43.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:40:43.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:40:43.972 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:40:44.008 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:44.008 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:44.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '03aec9f8-1b80-4c7c-ae43-862168b0b5e6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:44.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '03aec9f8-1b80-4c7c-ae43-862168b0b5e6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:44.351 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 03aec9f8-1b80-4c7c-ae43-862168b0b5e6 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:44.390 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:44.390 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:44.669 INFO:teuthology.orchestra.run.smithi137.stdout:[] 2020-07-23T12:40:44.670 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '03aec9f8-1b80-4c7c-ae43-862168b0b5e6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:44.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '03aec9f8-1b80-4c7c-ae43-862168b0b5e6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:44.671 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 03aec9f8-1b80-4c7c-ae43-862168b0b5e6 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:44.710 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:44.711 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:44.960 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2020-07-23T12:40:44.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '03aec9f8-1b80-4c7c-ae43-862168b0b5e6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:44.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '03aec9f8-1b80-4c7c-ae43-862168b0b5e6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:44.962 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 03aec9f8-1b80-4c7c-ae43-862168b0b5e6 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:45.001 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:45.001 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:45.266 INFO:teuthology.orchestra.run.smithi137.stdout:[] 2020-07-23T12:40:45.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '03aec9f8-1b80-4c7c-ae43-862168b0b5e6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:45.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '03aec9f8-1b80-4c7c-ae43-862168b0b5e6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:45.269 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 03aec9f8-1b80-4c7c-ae43-862168b0b5e6 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:45.307 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:45.307 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:45.539 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2020-07-23T12:40:45.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '30fc0b4a-5063-4d9f-a69b-5de0a471142a', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:45.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '30fc0b4a-5063-4d9f-a69b-5de0a471142a', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:45.541 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 30fc0b4a-5063-4d9f-a69b-5de0a471142a --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:45.581 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:45.581 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:45.843 INFO:teuthology.orchestra.run.smithi137.stdout:[] 2020-07-23T12:40:45.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '30fc0b4a-5063-4d9f-a69b-5de0a471142a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:45.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '30fc0b4a-5063-4d9f-a69b-5de0a471142a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:45.845 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 30fc0b4a-5063-4d9f-a69b-5de0a471142a --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:45.887 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:45.887 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:46.128 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2020-07-23T12:40:46.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '30fc0b4a-5063-4d9f-a69b-5de0a471142a', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:46.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '30fc0b4a-5063-4d9f-a69b-5de0a471142a', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:46.130 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 30fc0b4a-5063-4d9f-a69b-5de0a471142a --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:46.169 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:46.169 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:46.436 INFO:teuthology.orchestra.run.smithi137.stdout:[] 2020-07-23T12:40:46.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '30fc0b4a-5063-4d9f-a69b-5de0a471142a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:46.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '30fc0b4a-5063-4d9f-a69b-5de0a471142a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:46.438 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 30fc0b4a-5063-4d9f-a69b-5de0a471142a --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:46.476 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:46.477 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:46.718 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2020-07-23T12:40:46.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'a2b03b35-661e-4182-91b6-d60b3b65f694', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:46.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'a2b03b35-661e-4182-91b6-d60b3b65f694', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:46.720 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period a2b03b35-661e-4182-91b6-d60b3b65f694 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:46.758 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:46.758 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:47.016 INFO:teuthology.orchestra.run.smithi137.stdout:[] 2020-07-23T12:40:47.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'a2b03b35-661e-4182-91b6-d60b3b65f694', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:47.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'a2b03b35-661e-4182-91b6-d60b3b65f694', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:47.018 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period a2b03b35-661e-4182-91b6-d60b3b65f694 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:47.058 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:47.059 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:47.317 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2020-07-23T12:40:47.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'a2b03b35-661e-4182-91b6-d60b3b65f694', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:47.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'a2b03b35-661e-4182-91b6-d60b3b65f694', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:47.319 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period a2b03b35-661e-4182-91b6-d60b3b65f694 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:47.358 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:47.358 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:47.615 INFO:teuthology.orchestra.run.smithi137.stdout:[] 2020-07-23T12:40:47.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'a2b03b35-661e-4182-91b6-d60b3b65f694', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:47.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'a2b03b35-661e-4182-91b6-d60b3b65f694', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:47.617 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period a2b03b35-661e-4182-91b6-d60b3b65f694 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:47.660 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:47.660 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:47.905 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2020-07-23T12:40:47.906 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2020-07-23T12:40:47.907 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-46 2020-07-23T12:40:48.032 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-47 2020-07-23T12:40:48.713 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-48 2020-07-23T12:40:51.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:51.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:51.876 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:51.923 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:51.924 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:52.140 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507891.265297_92.1","last_update":"2020-07-23T12:38:11.265297Z"},{"marker":"1_1595507675.998941_262.2","last_update":"2020-07-23T12:34:35.998941Z"},{"marker":"1_1595508050.187532_291.1","last_update":"2020-07-23T12:40:50.187532Z"},{"marker":"1_1595507720.013682_138.1","last_update":"2020-07-23T12:35:20.013682Z"}] 2020-07-23T12:40:52.141 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-23T12:40:52.142 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:52.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:52.143 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:52.186 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:52.186 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:52.447 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":988729214735988972},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.265297_92.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.265297Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507675.998941_262.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:35.998941Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508050.187532_291.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:50.187532Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595507720.013682_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:20.013682Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:40:52.447 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-23T12:40:52.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:52.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:52.449 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:52.487 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:52.487 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:52.750 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507891.265297_92.1","last_update":"2020-07-23T12:38:11.265297Z"},{"marker":"1_1595507675.998941_262.2","last_update":"2020-07-23T12:34:35.998941Z"},{"marker":"1_1595508052.116906_294.1","last_update":"2020-07-23T12:40:52.116906Z"},{"marker":"1_1595507720.013682_138.1","last_update":"2020-07-23T12:35:20.013682Z"}] 2020-07-23T12:40:52.751 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-23T12:40:52.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:52.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:52.752 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:52.836 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:52.837 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:53.067 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":5372564157323408779},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.265297_92.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.265297Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507675.998941_262.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:35.998941Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508050.187532_291.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:50.187532Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595507720.013682_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:20.013682Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:40:53.068 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=2 master=1_1595508052.116906_294.1 target=1_1595508050.187532_291.1 2020-07-23T12:40:58.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:58.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:58.071 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:58.117 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:58.117 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:58.375 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":5372564157323408779},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.265297_92.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.265297Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507675.998941_262.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:35.998941Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508053.128510_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:53.128510Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595507720.013682_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:20.013682Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:40:58.376 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-23T12:40:58.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:58.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:58.378 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:58.458 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:58.458 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:58.739 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507891.265297_92.1","last_update":"2020-07-23T12:38:11.265297Z"},{"marker":"1_1595507675.998941_262.2","last_update":"2020-07-23T12:34:35.998941Z"},{"marker":"1_1595508053.128510_296.1","last_update":"2020-07-23T12:40:53.128510Z"},{"marker":"1_1595507720.013682_138.1","last_update":"2020-07-23T12:35:20.013682Z"}] 2020-07-23T12:40:58.740 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-23T12:40:58.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:58.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:58.742 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:58.787 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:58.787 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:59.067 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1496412237659215512},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.265297_92.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.265297Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507675.998941_262.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:35.998941Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508053.128510_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:53.128510Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595507720.013682_138.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:20.013682Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:40:59.067 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-23T12:40:59.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:59.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:59.070 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:59.147 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:59.147 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:59.392 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1595507891.299750_60.1","last_update":"2020-07-23T12:38:11.299750Z"},{"marker":"1_1595507676.941349_266.3","last_update":"2020-07-23T12:34:36.941349Z"},{"marker":"1_1595508059.093643_244.1","last_update":"2020-07-23T12:40:59.093643Z"},{"marker":"1_1595507720.947303_140.1","last_update":"2020-07-23T12:35:20.947303Z"}] 2020-07-23T12:40:59.392 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-23T12:40:59.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:59.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:40:59.396 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:40:59.437 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:40:59.438 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:40:59.699 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14254013310941169947},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.299750_60.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.299750Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507676.941349_266.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:36.941349Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508053.132845_242.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:53.132845Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595507720.947303_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:20.947303Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:40:59.699 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=2 master=1_1595508059.093643_244.1 target=1_1595508053.132845_242.1 2020-07-23T12:41:04.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:04.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:04.703 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:41:04.749 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:41:04.749 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:41:04.996 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14254013310941169947},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.299750_60.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.299750Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507676.941349_266.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:36.941349Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508053.132845_242.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:53.132845Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595507720.947303_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:20.947303Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:41:04.997 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=2 master=1_1595508059.093643_244.1 target=1_1595508053.132845_242.1 2020-07-23T12:41:09.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:10.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:10.001 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:41:10.036 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:41:10.037 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:41:10.304 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14254013310941169947},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.299750_60.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.299750Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507676.941349_266.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:36.941349Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508059.093643_244.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:59.093643Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595507720.947303_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:20.947303Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:41:10.304 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-23T12:41:10.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:10.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:10.306 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:41:10.345 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:41:10.346 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:41:10.596 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1595507891.299750_60.1","last_update":"2020-07-23T12:38:11.299750Z"},{"marker":"1_1595507676.941349_266.3","last_update":"2020-07-23T12:34:36.941349Z"},{"marker":"1_1595508059.093643_244.1","last_update":"2020-07-23T12:40:59.093643Z"},{"marker":"1_1595507720.947303_140.1","last_update":"2020-07-23T12:35:20.947303Z"}] 2020-07-23T12:41:10.596 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-23T12:41:10.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:10.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:10.598 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:41:10.641 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:41:10.642 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:41:10.885 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11610391562858473267},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.299750_60.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.299750Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507676.941349_266.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:36.941349Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508059.093643_244.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:59.093643Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595507720.947303_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:20.947303Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:41:10.886 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-23T12:41:10.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:10.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:10.888 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:41:10.967 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:41:10.967 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:41:11.184 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507890.675495_81.1","last_update":"2020-07-23T12:38:10.675495Z"},{"marker":"1_1595507676.033316_221.3","last_update":"2020-07-23T12:34:36.033316Z"},{"marker":"1_1595508052.115842_215.1","last_update":"2020-07-23T12:40:52.115842Z"},{"marker":"1_1595507720.041484_113.1","last_update":"2020-07-23T12:35:20.041484Z"}] 2020-07-23T12:41:11.185 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-23T12:41:11.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:11.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:11.186 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:41:11.274 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:41:11.274 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:41:11.543 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14010726936006743351},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507890.675495_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:10.675495Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507676.033316_221.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:36.033316Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508052.115842_215.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:52.115842Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595507720.041484_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:20.041484Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:41:11.543 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-23T12:41:11.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:11.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:11.545 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:41:11.626 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:41:11.627 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:41:11.892 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595507890.675495_81.1","last_update":"2020-07-23T12:38:10.675495Z"},{"marker":"1_1595507676.033316_221.3","last_update":"2020-07-23T12:34:36.033316Z"},{"marker":"1_1595508052.115842_215.1","last_update":"2020-07-23T12:40:52.115842Z"},{"marker":"1_1595507720.041484_113.1","last_update":"2020-07-23T12:35:20.041484Z"}] 2020-07-23T12:41:11.893 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-23T12:41:11.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:11.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:11.894 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:41:11.936 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:41:11.937 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:41:12.182 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3438718294491381619},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507890.675495_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:10.675495Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507676.033316_221.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:36.033316Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508052.115842_215.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:52.115842Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595507720.041484_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:35:20.041484Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:41:12.182 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-23T12:41:12.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:41:12.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:41:12.184 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:41:12.222 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:41:12.223 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:41:12.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:12.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:12.494 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:41:12.573 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:41:12.573 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:41:12.839 INFO:teuthology.orchestra.run.smithi137.stdout:[] 2020-07-23T12:41:12.840 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:41:12.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:41:12.841 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:41:12.884 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:41:12.884 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:41:13.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:13.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:13.143 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:41:13.224 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:41:13.224 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:41:13.482 INFO:teuthology.orchestra.run.smithi133.stdout:[] 2020-07-23T12:41:13.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:41:13.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:41:13.484 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:41:13.519 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:41:13.520 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:41:13.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:13.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:41:13.798 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:41:13.874 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:41:13.875 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:41:14.138 INFO:teuthology.orchestra.run.smithi137.stdout:[] 2020-07-23T12:41:14.139 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2020-07-23T12:41:14.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:41:14.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:41:14.143 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:41:14.182 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:41:14.183 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:41:14.211 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:41:14.208+0000 7efcff226080 0 failed reading obj info from .rgw.root:zone_info.0378667f-02f5-4ce6-824e-e2a396c12fc3: (2) No such file or directory 2020-07-23T12:41:14.211 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:41:14.208+0000 7efcff226080 0 WARNING: could not read zone params for zone id=0378667f-02f5-4ce6-824e-e2a396c12fc3 name=test-zone1 2020-07-23T12:41:14.213 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:41:14.208+0000 7efcff226080 0 failed reading obj info from .rgw.root:zone_info.41767e64-47c7-45a2-9984-76631257f2f6: (2) No such file or directory 2020-07-23T12:41:14.213 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:41:14.208+0000 7efcff226080 0 WARNING: could not read zone params for zone id=41767e64-47c7-45a2-9984-76631257f2f6 name=test-zone3 2020-07-23T12:41:14.230 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"T855BRGPSH14I4Y2","secret_key":"mMgfonT3dSbKAOArpyZfLcrPB3lPybUX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca"} 2020-07-23T12:41:14.231 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:41:14.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:41:14.232 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:41:14.322 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:41:14.322 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:41:14.587 INFO:teuthology.orchestra.run.smithi133.stderr:Sending period to new master zone 0378667f-02f5-4ce6-824e-e2a396c12fc3 2020-07-23T12:41:15.697 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","epoch":2,"predecessor_uuid":"30fc0b4a-5063-4d9f-a69b-5de0a471142a","sync_status":["","","","1_1595507963.420183_265.1"],"period_map":{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","zones":[{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"32a3ec76-95f5-4863-891c-667a67c89f48","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"41767e64-47c7-45a2-9984-76631257f2f6","name":"test-zone3","endpoints":["http://smithi137.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0378667f-02f5-4ce6-824e-e2a396c12fc3","val":68328169},{"key":"32a3ec76-95f5-4863-891c-667a67c89f48","val":3058828581},{"key":"41767e64-47c7-45a2-9984-76631257f2f6","val":2499065231},{"key":"42997773-3698-4448-bf7b-a00a5846d584","val":1484105665}]},"master_zonegroup":"test-zonegroup","master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","realm_name":"test-realm","realm_epoch":4} 2020-07-23T12:41:15.698 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2020-07-23T12:42:15.699 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-49 2020-07-23T12:42:15.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:15.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:15.809 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:42:15.854 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:42:15.855 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:42:16.054 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=a2b03b35-661e-4182-91b6-d60b3b65f694 2020-07-23T12:42:16.089 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595508135.749104_136.1","last_update":"2020-07-23T12:42:15.749104Z"}] 2020-07-23T12:42:16.090 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:42:16.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:16.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:16.093 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:42:16.134 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:42:16.135 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:42:16.381 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508049.480208_122.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:49.480208Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:42:16.382 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595508135.749104_136.1 target=1_1595508049.480208_122.1 2020-07-23T12:42:21.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:21.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:21.387 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:42:21.427 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:42:21.427 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:42:21.645 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508049.480208_122.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:49.480208Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:42:21.646 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595508135.749104_136.1 target=1_1595508049.480208_122.1 2020-07-23T12:42:26.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:26.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:26.651 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:42:26.698 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:42:26.698 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:42:26.934 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508049.480208_122.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:49.480208Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:42:26.934 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595508135.749104_136.1 target=1_1595508049.480208_122.1 2020-07-23T12:42:31.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:31.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:31.938 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:42:31.979 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:42:31.979 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:42:32.243 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508049.480208_122.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:49.480208Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:42:32.244 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595508135.749104_136.1 target=1_1595508049.480208_122.1 2020-07-23T12:42:37.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:37.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:37.247 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:42:37.287 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:42:37.287 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:42:37.534 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508049.480208_122.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:49.480208Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:42:37.534 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595508135.749104_136.1 target=1_1595508049.480208_122.1 2020-07-23T12:42:42.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:42.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:42.538 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:42:42.583 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:42:42.584 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:42:42.819 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508049.480208_122.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:49.480208Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:42:42.820 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595508135.749104_136.1 target=1_1595508049.480208_122.1 2020-07-23T12:42:46.826 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:42:46.820+0000 7fd09dff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:42:46.827 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:42:46.822+0000 7f4745ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:42:47.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:47.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:47.823 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:42:47.863 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:42:47.863 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:42:48.106 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508049.480208_122.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:49.480208Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:42:48.106 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595508135.749104_136.1 target=1_1595508049.480208_122.1 2020-07-23T12:42:53.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:53.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:53.109 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:42:53.150 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:42:53.150 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:42:53.410 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508049.480208_122.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:49.480208Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:42:53.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595508135.749104_136.1 target=1_1595508049.480208_122.1 2020-07-23T12:42:58.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:58.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:42:58.416 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:42:58.458 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:42:58.458 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:42:58.695 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508049.480208_122.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:49.480208Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:42:58.696 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595508135.749104_136.1 target=1_1595508049.480208_122.1 2020-07-23T12:43:03.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:43:03.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:43:03.700 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:43:03.739 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:43:03.740 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:43:03.977 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508049.480208_122.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:49.480208Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:43:03.978 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595508135.749104_136.1 target=1_1595508049.480208_122.1 2020-07-23T12:43:08.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:43:08.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:43:08.990 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:43:09.030 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:43:09.031 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:43:09.286 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508049.480208_122.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:49.480208Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:43:09.287 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595508135.749104_136.1 target=1_1595508049.480208_122.1 2020-07-23T12:43:14.289 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:43:14.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:43:14.290 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:43:14.330 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:43:14.331 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:43:14.578 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508049.480208_122.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:49.480208Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:43:14.579 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595508135.749104_136.1 target=1_1595508049.480208_122.1 2020-07-23T12:43:19.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:43:19.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:43:19.585 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:43:19.627 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:43:19.628 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:43:19.886 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508135.749104_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:42:15.749104Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:43:19.887 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:43:19.887 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:43:19.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:43:19.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:43:19.889 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:43:19.937 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:43:19.938 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:43:20.175 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508135.749104_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:42:15.749104Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:43:20.176 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:43:20.176 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:43:20.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:43:20.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:43:20.178 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:43:20.218 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:43:20.219 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:43:20.495 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508135.749104_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:42:15.749104Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:43:20.495 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:43:20.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=0378667f-02f5-4ce6-824e-e2a396c12fc3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:43:20.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=0378667f-02f5-4ce6-824e-e2a396c12fc3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:43:20.524 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=0378667f-02f5-4ce6-824e-e2a396c12fc3 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:43:20.576 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:43:20.576 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:43:20.607 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:43:20.601+0000 7f8c2643e080 0 failed reading obj info from .rgw.root:zone_info.0378667f-02f5-4ce6-824e-e2a396c12fc3: (2) No such file or directory 2020-07-23T12:43:20.607 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:43:20.601+0000 7f8c2643e080 0 WARNING: could not read zone params for zone id=0378667f-02f5-4ce6-824e-e2a396c12fc3 name=test-zone1 2020-07-23T12:43:20.608 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:43:20.601+0000 7f8c2643e080 0 failed reading obj info from .rgw.root:zone_info.41767e64-47c7-45a2-9984-76631257f2f6: (2) No such file or directory 2020-07-23T12:43:20.608 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:43:20.601+0000 7f8c2643e080 0 WARNING: could not read zone params for zone id=41767e64-47c7-45a2-9984-76631257f2f6 name=test-zone3 2020-07-23T12:43:20.623 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"T855BRGPSH14I4Y2","secret_key":"mMgfonT3dSbKAOArpyZfLcrPB3lPybUX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca"} 2020-07-23T12:43:20.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:43:20.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:43:20.625 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:43:20.702 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:43:20.703 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:43:20.965 INFO:teuthology.orchestra.run.smithi133.stderr:Sending period to new master zone 0378667f-02f5-4ce6-824e-e2a396c12fc3 2020-07-23T12:43:22.078 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","epoch":3,"predecessor_uuid":"30fc0b4a-5063-4d9f-a69b-5de0a471142a","sync_status":["","","","1_1595507963.420183_265.1"],"period_map":{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","zones":[{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"32a3ec76-95f5-4863-891c-667a67c89f48","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"41767e64-47c7-45a2-9984-76631257f2f6","name":"test-zone3","endpoints":["http://smithi137.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3"}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0378667f-02f5-4ce6-824e-e2a396c12fc3","val":68328169},{"key":"32a3ec76-95f5-4863-891c-667a67c89f48","val":3058828581},{"key":"41767e64-47c7-45a2-9984-76631257f2f6","val":2499065231},{"key":"42997773-3698-4448-bf7b-a00a5846d584","val":1484105665}]},"master_zonegroup":"test-zonegroup","master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","realm_name":"test-realm","realm_epoch":4} 2020-07-23T12:43:22.079 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "0378667f-02f5-4ce6-824e-e2a396c12fc3" 2020-07-23T12:44:22.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:44:22.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:44:22.426 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:44:22.469 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:44:22.469 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:44:22.496 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:44:22.488+0000 7fa782c6e080 0 failed reading obj info from .rgw.root:zone_info.0378667f-02f5-4ce6-824e-e2a396c12fc3: (2) No such file or directory 2020-07-23T12:44:22.496 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:44:22.488+0000 7fa782c6e080 0 WARNING: could not read zone params for zone id=0378667f-02f5-4ce6-824e-e2a396c12fc3 name=test-zone1 2020-07-23T12:44:22.497 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:44:22.488+0000 7fa782c6e080 0 failed reading obj info from .rgw.root:zone_info.41767e64-47c7-45a2-9984-76631257f2f6: (2) No such file or directory 2020-07-23T12:44:22.498 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:44:22.488+0000 7fa782c6e080 0 WARNING: could not read zone params for zone id=41767e64-47c7-45a2-9984-76631257f2f6 name=test-zone3 2020-07-23T12:44:22.514 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"T855BRGPSH14I4Y2","secret_key":"mMgfonT3dSbKAOArpyZfLcrPB3lPybUX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca"} 2020-07-23T12:44:22.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:44:22.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:44:22.517 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:44:22.596 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:44:22.596 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:44:22.862 INFO:teuthology.orchestra.run.smithi133.stderr:Sending period to new master zone 0378667f-02f5-4ce6-824e-e2a396c12fc3 2020-07-23T12:44:23.978 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","epoch":4,"predecessor_uuid":"30fc0b4a-5063-4d9f-a69b-5de0a471142a","sync_status":["","","","1_1595507963.420183_265.1"],"period_map":{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","zones":[{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"32a3ec76-95f5-4863-891c-667a67c89f48","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"41767e64-47c7-45a2-9984-76631257f2f6","name":"test-zone3","endpoints":["http://smithi137.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3"}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0378667f-02f5-4ce6-824e-e2a396c12fc3","val":68328169},{"key":"32a3ec76-95f5-4863-891c-667a67c89f48","val":3058828581},{"key":"41767e64-47c7-45a2-9984-76631257f2f6","val":2499065231},{"key":"42997773-3698-4448-bf7b-a00a5846d584","val":1484105665}]},"master_zonegroup":"test-zonegroup","master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","realm_name":"test-realm","realm_epoch":4} 2020-07-23T12:44:23.979 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2020-07-23T12:45:23.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:45:23.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:45:23.997 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:45:24.050 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:45:24.051 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:45:24.071 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:45:24.063+0000 7f4dc3510080 0 failed reading obj info from .rgw.root:zone_info.0378667f-02f5-4ce6-824e-e2a396c12fc3: (2) No such file or directory 2020-07-23T12:45:24.071 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:45:24.063+0000 7f4dc3510080 0 WARNING: could not read zone params for zone id=0378667f-02f5-4ce6-824e-e2a396c12fc3 name=test-zone1 2020-07-23T12:45:24.072 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:45:24.067+0000 7f4dc3510080 0 failed reading obj info from .rgw.root:zone_info.41767e64-47c7-45a2-9984-76631257f2f6: (2) No such file or directory 2020-07-23T12:45:24.072 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:45:24.067+0000 7f4dc3510080 0 WARNING: could not read zone params for zone id=41767e64-47c7-45a2-9984-76631257f2f6 name=test-zone3 2020-07-23T12:45:24.093 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"T855BRGPSH14I4Y2","secret_key":"mMgfonT3dSbKAOArpyZfLcrPB3lPybUX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca"} 2020-07-23T12:45:24.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:45:24.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:45:24.095 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:45:24.170 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:45:24.171 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:45:24.433 INFO:teuthology.orchestra.run.smithi133.stderr:Sending period to new master zone 0378667f-02f5-4ce6-824e-e2a396c12fc3 2020-07-23T12:45:25.534 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","epoch":5,"predecessor_uuid":"30fc0b4a-5063-4d9f-a69b-5de0a471142a","sync_status":["","","","1_1595507963.420183_265.1"],"period_map":{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","zones":[{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"32a3ec76-95f5-4863-891c-667a67c89f48","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"41767e64-47c7-45a2-9984-76631257f2f6","name":"test-zone3","endpoints":["http://smithi137.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0378667f-02f5-4ce6-824e-e2a396c12fc3","val":68328169},{"key":"32a3ec76-95f5-4863-891c-667a67c89f48","val":3058828581},{"key":"41767e64-47c7-45a2-9984-76631257f2f6","val":2499065231},{"key":"42997773-3698-4448-bf7b-a00a5846d584","val":1484105665}]},"master_zonegroup":"test-zonegroup","master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","realm_name":"test-realm","realm_epoch":4} 2020-07-23T12:45:25.535 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2020-07-23T12:46:25.544 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2020-07-23T12:46:25.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:25.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:25.585 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:46:25.628 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:25.628 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:25.842 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=a2b03b35-661e-4182-91b6-d60b3b65f694 2020-07-23T12:46:25.884 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595508135.749104_136.1","last_update":"2020-07-23T12:42:15.749104Z"}] 2020-07-23T12:46:25.884 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:46:25.885 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:25.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:25.891 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:46:25.936 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:25.937 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:26.192 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508135.749104_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:42:15.749104Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:46:26.193 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:46:26.194 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:46:26.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:26.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:26.199 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:46:26.240 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:26.241 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:26.502 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508135.749104_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:42:15.749104Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:46:26.503 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:46:26.504 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:46:26.504 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:26.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:26.505 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:46:26.550 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:26.550 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:26.806 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508135.749104_136.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:42:15.749104Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:46:26.807 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:46:26.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:26.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'T855BRGPSH14I4Y2', '--secret', 'mMgfonT3dSbKAOArpyZfLcrPB3lPybUX', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:26.809 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key T855BRGPSH14I4Y2 --secret mMgfonT3dSbKAOArpyZfLcrPB3lPybUX --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:46:26.902 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:26.902 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:26.970 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:46:26.962+0000 7fe2aab74080 0 failed reading obj info from .rgw.root:zone_info.0378667f-02f5-4ce6-824e-e2a396c12fc3: (2) No such file or directory 2020-07-23T12:46:26.971 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:46:26.962+0000 7fe2aab74080 0 WARNING: could not read zone params for zone id=0378667f-02f5-4ce6-824e-e2a396c12fc3 name=test-zone1 2020-07-23T12:46:26.972 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:46:26.966+0000 7fe2aab74080 0 failed reading obj info from .rgw.root:zone_info.41767e64-47c7-45a2-9984-76631257f2f6: (2) No such file or directory 2020-07-23T12:46:26.972 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:46:26.966+0000 7fe2aab74080 0 WARNING: could not read zone params for zone id=41767e64-47c7-45a2-9984-76631257f2f6 name=test-zone3 2020-07-23T12:46:26.988 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"3092881a-a3b3-44ca-a502-eb7bbee7bf12","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"T855BRGPSH14I4Y2","secret_key":"mMgfonT3dSbKAOArpyZfLcrPB3lPybUX"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca"} 2020-07-23T12:46:26.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:26.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:26.990 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:46:27.069 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:27.070 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:34.971 INFO:teuthology.orchestra.run.smithi133.stderr:Sending period to new master zone 0378667f-02f5-4ce6-824e-e2a396c12fc3 2020-07-23T12:46:36.073 INFO:teuthology.orchestra.run.smithi133.stdout:{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","epoch":6,"predecessor_uuid":"30fc0b4a-5063-4d9f-a69b-5de0a471142a","sync_status":["","","","1_1595507963.420183_265.1"],"period_map":{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","zones":[{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"3092881a-a3b3-44ca-a502-eb7bbee7bf12","name":"remove","endpoints":[],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"32a3ec76-95f5-4863-891c-667a67c89f48","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"41767e64-47c7-45a2-9984-76631257f2f6","name":"test-zone3","endpoints":["http://smithi137.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0378667f-02f5-4ce6-824e-e2a396c12fc3","val":68328169},{"key":"3092881a-a3b3-44ca-a502-eb7bbee7bf12","val":3127035531},{"key":"32a3ec76-95f5-4863-891c-667a67c89f48","val":3058828581},{"key":"41767e64-47c7-45a2-9984-76631257f2f6","val":2499065231},{"key":"42997773-3698-4448-bf7b-a00a5846d584","val":1484105665}]},"master_zonegroup":"test-zonegroup","master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","realm_name":"test-realm","realm_epoch":4} 2020-07-23T12:46:36.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:36.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:36.079 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:46:36.128 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:36.129 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:36.158 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:46:36.144+0000 7fccfc996080 0 failed reading obj info from .rgw.root:zone_info.32a3ec76-95f5-4863-891c-667a67c89f48: (2) No such file or directory 2020-07-23T12:46:36.158 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:46:36.144+0000 7fccfc996080 0 WARNING: could not read zone params for zone id=32a3ec76-95f5-4863-891c-667a67c89f48 name=test-zone4 2020-07-23T12:46:36.159 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:46:36.148+0000 7fccfc996080 0 failed reading obj info from .rgw.root:zone_info.42997773-3698-4448-bf7b-a00a5846d584: (2) No such file or directory 2020-07-23T12:46:36.159 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:46:36.148+0000 7fccfc996080 0 WARNING: could not read zone params for zone id=42997773-3698-4448-bf7b-a00a5846d584 name=test-zone2 2020-07-23T12:46:36.160 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","zones":[{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"32a3ec76-95f5-4863-891c-667a67c89f48","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"41767e64-47c7-45a2-9984-76631257f2f6","name":"test-zone3","endpoints":["http://smithi137.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}} 2020-07-23T12:46:36.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:36.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:36.170 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:46:36.237 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:36.238 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:36.255 INFO:teuthology.orchestra.run.smithi137.stderr:zone name remove not found in zonegroup test-zonegroup 2020-07-23T12:46:36.261 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-07-23T12:46:36.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:36.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:36.265 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2020-07-23T12:46:36.303 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:36.303 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:36.347 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:46:36.342+0000 7f6b01172080 0 failed reading obj info from .rgw.root:zone_info.0378667f-02f5-4ce6-824e-e2a396c12fc3: (2) No such file or directory 2020-07-23T12:46:36.347 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:46:36.342+0000 7f6b01172080 0 WARNING: could not read zone params for zone id=0378667f-02f5-4ce6-824e-e2a396c12fc3 name=test-zone1 2020-07-23T12:46:36.349 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:46:36.342+0000 7f6b01172080 0 failed reading obj info from .rgw.root:zone_info.41767e64-47c7-45a2-9984-76631257f2f6: (2) No such file or directory 2020-07-23T12:46:36.349 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:46:36.342+0000 7f6b01172080 0 WARNING: could not read zone params for zone id=41767e64-47c7-45a2-9984-76631257f2f6 name=test-zone3 2020-07-23T12:46:36.357 INFO:teuthology.orchestra.run.smithi133.stderr:2020-07-23T12:46:36.350+0000 7f6b01172080 0 zone id 3092881a-a3b3-44ca-a502-eb7bbee7bf12 is not a part of zonegroup default 2020-07-23T12:46:36.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:36.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:36.367 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:46:36.407 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:36.408 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:36.470 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:46:36.464+0000 7fb107f4d080 0 WARNING: can't generate connection for zone 3092881a-a3b3-44ca-a502-eb7bbee7bf12 id remove: no endpoints defined 2020-07-23T12:46:36.738 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:46:36.728+0000 7fcfc2ffd700 -1 failed to decode period 2020-07-23T12:46:36.739 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:46:36.726+0000 7f46577fe700 -1 failed to decode period 2020-07-23T12:46:36.739 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:46:36.726+0000 7f00beffd700 -1 failed to decode period 2020-07-23T12:46:36.762 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","epoch":7,"predecessor_uuid":"30fc0b4a-5063-4d9f-a69b-5de0a471142a","sync_status":["","","","1_1595507963.420183_265.1"],"period_map":{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","zones":[{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"32a3ec76-95f5-4863-891c-667a67c89f48","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"41767e64-47c7-45a2-9984-76631257f2f6","name":"test-zone3","endpoints":["http://smithi137.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0378667f-02f5-4ce6-824e-e2a396c12fc3","val":68328169},{"key":"32a3ec76-95f5-4863-891c-667a67c89f48","val":3058828581},{"key":"41767e64-47c7-45a2-9984-76631257f2f6","val":2499065231},{"key":"42997773-3698-4448-bf7b-a00a5846d584","val":1484105665}]},"master_zonegroup":"test-zonegroup","master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","realm_name":"test-realm","realm_epoch":4} 2020-07-23T12:46:36.764 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2020-07-23T12:46:36.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:36.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:36.766 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:46:36.846 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:36.847 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:36.879 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"d86738b0-96e6-44aa-b057-1c2a2479f53b","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}} 2020-07-23T12:46:36.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:36.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:36.882 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:46:36.959 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:36.960 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:37.042 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"0710c6db-8e69-4fbc-a6e9-29e6e593ec6e","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca"} 2020-07-23T12:46:37.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:37.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:37.044 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:46:37.129 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:37.130 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:37.478 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","epoch":8,"predecessor_uuid":"30fc0b4a-5063-4d9f-a69b-5de0a471142a","sync_status":["","","","1_1595507963.420183_265.1"],"period_map":{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","zonegroups":[{"id":"d86738b0-96e6-44aa-b057-1c2a2479f53b","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"0710c6db-8e69-4fbc-a6e9-29e6e593ec6e","zones":[{"id":"0710c6db-8e69-4fbc-a6e9-29e6e593ec6e","name":"remove","endpoints":[],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","zones":[{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"32a3ec76-95f5-4863-891c-667a67c89f48","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"41767e64-47c7-45a2-9984-76631257f2f6","name":"test-zone3","endpoints":["http://smithi137.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0378667f-02f5-4ce6-824e-e2a396c12fc3","val":68328169},{"key":"0710c6db-8e69-4fbc-a6e9-29e6e593ec6e","val":3403953694},{"key":"32a3ec76-95f5-4863-891c-667a67c89f48","val":3058828581},{"key":"41767e64-47c7-45a2-9984-76631257f2f6","val":2499065231},{"key":"42997773-3698-4448-bf7b-a00a5846d584","val":1484105665}]},"master_zonegroup":"test-zonegroup","master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","realm_name":"test-realm","realm_epoch":4} 2020-07-23T12:46:37.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:37.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:37.480 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:46:37.560 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:37.560 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:37.603 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:46:37.598+0000 7f00beffd700 -1 failed to decode period 2020-07-23T12:46:37.603 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:46:37.598+0000 7f46577fe700 -1 failed to decode period 2020-07-23T12:46:37.631 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:46:37.624+0000 7fa1ff11e080 0 zone id 0710c6db-8e69-4fbc-a6e9-29e6e593ec6e is not a part of zonegroup test-zonegroup 2020-07-23T12:46:37.632 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:46:37.624+0000 7fa1ff11e080 0 zone id 0710c6db-8e69-4fbc-a6e9-29e6e593ec6e is not a part of zonegroup default 2020-07-23T12:46:37.632 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:46:37.628+0000 7fcfc2ffd700 -1 failed to decode period 2020-07-23T12:46:37.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:37.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:37.658 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:46:37.738 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:37.738 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:37.790 INFO:teuthology.orchestra.run.smithi137.stderr:2020-07-23T12:46:37.788+0000 7fe7453f1080 0 ERROR: zonegroup remove_zg has a non existent master zone 2020-07-23T12:46:37.801 DEBUG:teuthology.orchestra.run:got remote process result: 22 2020-07-23T12:46:37.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:37.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:37.803 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:46:37.878 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:37.879 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:37.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:37.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:37.923 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:46:37.992 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:37.993 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:38.291 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:46:38.288+0000 7fcf98ff9700 -1 failed to decode period 2020-07-23T12:46:38.292 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:46:38.286+0000 7f00beffd700 -1 failed to decode period 2020-07-23T12:46:38.293 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:46:38.286+0000 7f46577fe700 -1 failed to decode period 2020-07-23T12:46:38.337 INFO:teuthology.orchestra.run.smithi137.stdout:{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","epoch":9,"predecessor_uuid":"30fc0b4a-5063-4d9f-a69b-5de0a471142a","sync_status":["","","","1_1595507963.420183_265.1"],"period_map":{"id":"a2b03b35-661e-4182-91b6-d60b3b65f694","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","zones":[{"id":"0378667f-02f5-4ce6-824e-e2a396c12fc3","name":"test-zone1","endpoints":["http://smithi137.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"32a3ec76-95f5-4863-891c-667a67c89f48","name":"test-zone4","endpoints":["http://smithi133.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"41767e64-47c7-45a2-9984-76631257f2f6","name":"test-zone3","endpoints":["http://smithi137.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"42997773-3698-4448-bf7b-a00a5846d584","name":"test-zone2","endpoints":["http://smithi133.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0378667f-02f5-4ce6-824e-e2a396c12fc3","val":68328169},{"key":"32a3ec76-95f5-4863-891c-667a67c89f48","val":3058828581},{"key":"41767e64-47c7-45a2-9984-76631257f2f6","val":2499065231},{"key":"42997773-3698-4448-bf7b-a00a5846d584","val":1484105665}]},"master_zonegroup":"test-zonegroup","master_zone":"0378667f-02f5-4ce6-824e-e2a396c12fc3","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"bca7b68d-77da-418f-a5f9-ecb5e2e483ca","realm_name":"test-realm","realm_epoch":4} 2020-07-23T12:46:38.338 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2020-07-23T12:46:38.339 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-50 2020-07-23T12:46:38.894 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-51 2020-07-23T12:46:38.993 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-52 2020-07-23T12:46:39.099 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-53 2020-07-23T12:46:39.155 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-54 2020-07-23T12:46:39.240 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-55 2020-07-23T12:46:43.250 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2020-07-23T12:46:43.251 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-56 2020-07-23T12:46:43.909 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-57 2020-07-23T12:46:44.810 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-58 2020-07-23T12:46:45.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'pdivcf-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:45.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'pdivcf-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:45.652 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket pdivcf-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:46:45.688 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:45.688 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:46.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'pdivcf-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:46.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'pdivcf-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:46.058 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket pdivcf-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:46:46.138 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:46.139 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:46.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'pdivcf-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:46.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'pdivcf-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:46:46.493 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket pdivcf-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:46:46.574 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:46.575 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:46.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:46.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:46.930 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:46:47.011 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:47.012 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:47.278 INFO:teuthology.orchestra.run.smithi137.stdout:["obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:3","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28:8","obj_delete_at_hint.0000000026","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29:8","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:3","obj_delete_at_hint.0000000028","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7335.1-pdivcf-49","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:7","obj_delete_at_hint.0000000040","meta.full-sync.index.0","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28-pdivcf-19","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28:8","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35:8","data.full-sync.index.41767e64-47c7-45a2-9984-76631257f2f6.0","obj_delete_at_hint.0000000024","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.5-pdivcf-54","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:1","data_log.3","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:1","data.full-sync.index.42997773-3698-4448-bf7b-a00a5846d584.2","2020-07-23-12-42997773-3698-4448-bf7b-a00a5846d584.6302.1-pdivcf-42","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:1","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-15:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24:1","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-37:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46:10","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19-pdivcf-10","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","2020-07-23-12-42997773-3698-4448-bf7b-a00a5846d584.6302.2-pdivcf-43","obj_delete_at_hint.0000000058","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:3","obj_delete_at_hint.0000000052","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.16-pdivcf-7","datalog.sync-status.shard.41767e64-47c7-45a2-9984-76631257f2f6.0.retry","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.42-pdivcf-33","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29-pdivcf-20","obj_delete_at_hint.0000000110","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23-pdivcf-14","obj_delete_at_hint.0000000096","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-13:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22:1","obj_delete_at_hint.0000000087","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:8","obj_delete_at_hint.0000000008","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33-pdivcf-24","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:7","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27-pdivcf-18","obj_delete_at_hint.0000000006","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:3","obj_delete_at_hint.0000000029","datalog.sync-status.42997773-3698-4448-bf7b-a00a5846d584","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45-pdivcf-36","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25-pdivcf-16","bilog.trim","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:8","obj_delete_at_hint.0000000016","data.full-sync.index.41767e64-47c7-45a2-9984-76631257f2f6.2","obj_delete_at_hint.0000000041","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:5","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37-pdivcf-28","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:5","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31:8","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.7-pdivcf-56","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.5-pdivcf-5","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.1-pdivcf-44","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-38:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47:10","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-48:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.5:8","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-48:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.5:8","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-46:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.3:8","data.full-sync.index.42997773-3698-4448-bf7b-a00a5846d584.1","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.9-pdivcf-58","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26-pdivcf-17","obj_delete_at_hint.0000000121","data.full-sync.index.41767e64-47c7-45a2-9984-76631257f2f6.3","data_log.1","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:1","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-37:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46:10","meta.full-sync.index.1","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:8","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.2-pdivcf-2","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-13:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22:1","obj_delete_at_hint.0000000077","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28:5","obj_delete_at_hint.0000000032","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29:8","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:3","datalog.sync-status.shard.41767e64-47c7-45a2-9984-76631257f2f6.2.retry","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-47:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.4:8","obj_delete_at_hint.0000000065","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-36:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45:10","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.2-pdivcf-51","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:1","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22-pdivcf-13","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21-pdivcf-12","mdlog.sync-status.shard.2","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:7","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:3","obj_delete_at_hint.0000000045","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38-pdivcf-29","obj_delete_at_hint.0000000002","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.17-pdivcf-8","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.2-pdivcf-45","obj_delete_at_hint.0000000116","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:1","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:1","obj_delete_at_hint.0000000034","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:1","datalog.sync-status.shard.41767e64-47c7-45a2-9984-76631257f2f6.0","obj_delete_at_hint.0000000101","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.6-pdivcf-6","datalog.sync-status.shard.41767e64-47c7-45a2-9984-76631257f2f6.2","mdlog.sync-status.shard.1","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:5","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","meta.log.a2b03b35-661e-4182-91b6-d60b3b65f694.3","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35-pdivcf-26","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","datalog.sync-status.shard.42997773-3698-4448-bf7b-a00a5846d584.1","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.3-pdivcf-3","meta.full-sync.index.3","obj_delete_at_hint.0000000118","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:1","data.full-sync.index.42997773-3698-4448-bf7b-a00a5846d584.0","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35:8","obj_delete_at_hint.0000000036","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34:8","obj_delete_at_hint.0000000103","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-38:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47:10","mdlog.sync-status.shard.3","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34:8","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:3","mdlog.sync-status.shard.0","datalog.sync-status.shard.41767e64-47c7-45a2-9984-76631257f2f6.3","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:3","datalog.sync-status.shard.42997773-3698-4448-bf7b-a00a5846d584.3","obj_delete_at_hint.0000000039","datalog.sync-status.shard.41767e64-47c7-45a2-9984-76631257f2f6.3.retry","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:8","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.1-pdivcf-1","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:8","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.39-pdivcf-30","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.48-pdivcf-39","obj_delete_at_hint.0000000100","data.full-sync.index.41767e64-47c7-45a2-9984-76631257f2f6.1","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-14:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23:1","obj_delete_at_hint.0000000093","datalog.sync-status.shard.42997773-3698-4448-bf7b-a00a5846d584.1.retry","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.6-pdivcf-55","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.18-pdivcf-9","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24-pdivcf-15","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.50-pdivcf-41","data_log.0","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.49-pdivcf-40","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:7","obj_delete_at_hint.0000000012","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31:8","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-46:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.3:8","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:3","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30-pdivcf-21","datalog.sync-status.shard.42997773-3698-4448-bf7b-a00a5846d584.2.retry","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:5","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:3","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.4-pdivcf-47","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.8-pdivcf-57","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.1-pdivcf-50","datalog.sync-status.41767e64-47c7-45a2-9984-76631257f2f6","obj_delete_at_hint.0000000072","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28:5","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","sync.error-log.1","obj_delete_at_hint.0000000033","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47-pdivcf-38","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.36-pdivcf-27","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-36:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45:10","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:7","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44-pdivcf-35","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:1","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31-pdivcf-22","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.4-pdivcf-53","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.3-pdivcf-46","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","datalog.sync-status.shard.41767e64-47c7-45a2-9984-76631257f2f6.1.retry","obj_delete_at_hint.0000000013","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34-pdivcf-25","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32:8","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46-pdivcf-37","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-49:0378667f-02f5-4ce6-824e-e2a396c12fc3.7335.1:5","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20-pdivcf-11","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-14:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23:1","obj_delete_at_hint.0000000023","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-47:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.4:8","datalog.sync-status.shard.42997773-3698-4448-bf7b-a00a5846d584.3.retry","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32-pdivcf-23","obj_delete_at_hint.0000000107","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.5-pdivcf-48","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:8","obj_delete_at_hint.0000000046","meta.full-sync.index.2","datalog.sync-status.shard.42997773-3698-4448-bf7b-a00a5846d584.2","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.4-pdivcf-4","data_log.2","obj_delete_at_hint.0000000067","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40-pdivcf-31","datalog.sync-status.shard.41767e64-47c7-45a2-9984-76631257f2f6.1","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:1","obj_delete_at_hint.0000000086","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43-pdivcf-34","data.full-sync.index.42997773-3698-4448-bf7b-a00a5846d584.3","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:7","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.3-pdivcf-52","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:5","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33:8","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33:8","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:3","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","datalog.sync-status.shard.42997773-3698-4448-bf7b-a00a5846d584.0","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41-pdivcf-32","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-15:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24:1","obj_delete_at_hint.0000000124","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:1","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32:8","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30:8","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-49:0378667f-02f5-4ce6-824e-e2a396c12fc3.7335.1:5","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30:8","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:5"] 2020-07-23T12:46:47.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:47.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:47.312 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:46:47.348 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:47.349 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:47.608 INFO:teuthology.orchestra.run.smithi133.stdout:["obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:3","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28:8","obj_delete_at_hint.0000000026","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35:8","obj_delete_at_hint.0000000028","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7335.1-pdivcf-49","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:7","obj_delete_at_hint.0000000040","meta.full-sync.index.0","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28-pdivcf-19","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-47:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.4:8","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:3","data.full-sync.index.41767e64-47c7-45a2-9984-76631257f2f6.0","obj_delete_at_hint.0000000024","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.5-pdivcf-54","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:8","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:1","data_log.3","2020-07-23-12-42997773-3698-4448-bf7b-a00a5846d584.6302.1-pdivcf-42","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:1","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-15:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24:1","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19-pdivcf-10","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:3","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","2020-07-23-12-42997773-3698-4448-bf7b-a00a5846d584.6302.2-pdivcf-43","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28:5","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.16-pdivcf-7","datalog.sync-status.shard.41767e64-47c7-45a2-9984-76631257f2f6.0.retry","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29-pdivcf-20","obj_delete_at_hint.0000000110","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23-pdivcf-14","obj_delete_at_hint.0000000096","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.0","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-13:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22:1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-37:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46:10","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:8","obj_delete_at_hint.0000000087","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:8","obj_delete_at_hint.0000000008","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33-pdivcf-24","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27-pdivcf-18","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45-pdivcf-36","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25-pdivcf-16","bilog.trim","obj_delete_at_hint.0000000016","data.full-sync.index.41767e64-47c7-45a2-9984-76631257f2f6.2","obj_delete_at_hint.0000000041","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.37-pdivcf-28","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31:8","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.5-pdivcf-5","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.1-pdivcf-44","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-38:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47:10","obj_delete_at_hint.0000000014","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:1","obj_delete_at_hint.0000000112","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-48:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.5:8","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26-pdivcf-17","obj_delete_at_hint.0000000121","data.full-sync.index.41767e64-47c7-45a2-9984-76631257f2f6.3","data_log.1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33:8","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:1","obj_delete_at_hint.0000000125","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:3","obj_delete_at_hint.0000000082","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-37:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46:10","meta.full-sync.index.1","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.2-pdivcf-2","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:1","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29:8","datalog.sync-status.shard.41767e64-47c7-45a2-9984-76631257f2f6.2.retry","obj_delete_at_hint.0000000053","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:7","obj_delete_at_hint.0000000091","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-47:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.4:8","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.2-pdivcf-51","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:1","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22-pdivcf-13","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21-pdivcf-12","mdlog.sync-status.shard.2","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.0.retry","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.3","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:7","obj_delete_at_hint.0000000045","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.3.retry","obj_delete_at_hint.0000000002","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:1","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.17-pdivcf-8","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.2-pdivcf-45","obj_delete_at_hint.0000000116","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-13:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22:1","obj_delete_at_hint.0000000034","datalog.sync-status.shard.41767e64-47c7-45a2-9984-76631257f2f6.0","obj_delete_at_hint.0000000101","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.6-pdivcf-6","datalog.sync-status.shard.41767e64-47c7-45a2-9984-76631257f2f6.2","mdlog.sync-status.shard.1","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:5","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28:8","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","meta.log.a2b03b35-661e-4182-91b6-d60b3b65f694.3","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35-pdivcf-26","obj_delete_at_hint.0000000066","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:5","obj_delete_at_hint.0000000068","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:1","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.3-pdivcf-3","data.full-sync.index.0378667f-02f5-4ce6-824e-e2a396c12fc3.0","meta.full-sync.index.3","obj_delete_at_hint.0000000118","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:3","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35:8","obj_delete_at_hint.0000000036","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:5","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34:8","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","data.full-sync.index.0378667f-02f5-4ce6-824e-e2a396c12fc3.2","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:3","mdlog.sync-status.shard.0","datalog.sync-status.shard.41767e64-47c7-45a2-9984-76631257f2f6.3","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:3","obj_delete_at_hint.0000000039","datalog.sync-status.shard.41767e64-47c7-45a2-9984-76631257f2f6.3.retry","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:8","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34:8","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-36:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45:10","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.1-pdivcf-1","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:8","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.48-pdivcf-39","obj_delete_at_hint.0000000100","data.full-sync.index.41767e64-47c7-45a2-9984-76631257f2f6.1","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-14:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23:1","obj_delete_at_hint.0000000093","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.18-pdivcf-9","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24-pdivcf-15","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.50-pdivcf-41","data_log.0","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.49-pdivcf-40","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:7","obj_delete_at_hint.0000000012","data.full-sync.index.0378667f-02f5-4ce6-824e-e2a396c12fc3.3","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-46:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.3:8","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30-pdivcf-21","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-48:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.5:8","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:5","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:3","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.4-pdivcf-47","obj_delete_at_hint.0000000111","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29:8","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.8-pdivcf-57","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","datalog.sync-status.41767e64-47c7-45a2-9984-76631257f2f6","obj_delete_at_hint.0000000072","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32:8","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28:5","obj_delete_at_hint.0000000062","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31:8","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.2.retry","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47-pdivcf-38","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-36:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45:10","obj_delete_at_hint.0000000025","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:5","obj_delete_at_hint.0000000123","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.2","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31-pdivcf-22","obj_delete_at_hint.0000000108","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.1.retry","obj_delete_at_hint.0000000094","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.3-pdivcf-46","obj_delete_at_hint.0000000060","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-14:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23:1","obj_delete_at_hint.0000000109","datalog.sync-status.shard.41767e64-47c7-45a2-9984-76631257f2f6.1.retry","obj_delete_at_hint.0000000013","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34-pdivcf-25","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:7","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46-pdivcf-37","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20-pdivcf-11","obj_delete_at_hint.0000000023","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:8","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32-pdivcf-23","obj_delete_at_hint.0000000107","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.5-pdivcf-48","obj_delete_at_hint.0000000046","meta.full-sync.index.2","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.4-pdivcf-4","data_log.2","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-49:0378667f-02f5-4ce6-824e-e2a396c12fc3.7335.1:5","obj_delete_at_hint.0000000067","data.full-sync.index.0378667f-02f5-4ce6-824e-e2a396c12fc3.1","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.40-pdivcf-31","datalog.sync-status.shard.41767e64-47c7-45a2-9984-76631257f2f6.1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-38:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47:10","obj_delete_at_hint.0000000086","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.43-pdivcf-34","datalog.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:1","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:5","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33:8","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:3","obj_delete_at_hint.0000000050","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-15:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24:1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:3","obj_delete_at_hint.0000000102","obj_delete_at_hint.0000000124","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:3","obj_delete_at_hint.0000000001","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:7","obj_delete_at_hint.0000000126","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30:8","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32:8","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-49:0378667f-02f5-4ce6-824e-e2a396c12fc3.7335.1:5","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-46:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.3:8","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","bucket.sync-status.41767e64-47c7-45a2-9984-76631257f2f6:pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30:8"] 2020-07-23T12:46:47.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:47.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:47.642 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:46:47.683 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:47.684 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:47.995 INFO:teuthology.orchestra.run.smithi137.stdout:["obj_delete_at_hint.0000000078","meta.history","meta.log.03aec9f8-1b80-4c7c-ae43-862168b0b5e6.2","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29:8","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:1","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:3","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35:8","obj_delete_at_hint.0000000028","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7335.1-pdivcf-49","obj_delete_at_hint.0000000040","meta.full-sync.index.0","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28-pdivcf-19","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28:8","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-47:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.4:8","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35:8","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:3","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:8","data_log.3","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:1","data.full-sync.index.42997773-3698-4448-bf7b-a00a5846d584.2","2020-07-23-12-42997773-3698-4448-bf7b-a00a5846d584.6302.1-pdivcf-42","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-37:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46:10","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19-pdivcf-10","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:3","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","2020-07-23-12-42997773-3698-4448-bf7b-a00a5846d584.6302.2-pdivcf-43","obj_delete_at_hint.0000000058","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:3","obj_delete_at_hint.0000000052","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28:5","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.16-pdivcf-7","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29-pdivcf-20","obj_delete_at_hint.0000000110","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23-pdivcf-14","obj_delete_at_hint.0000000096","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.0","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-37:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46:10","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:8","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33-pdivcf-24","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:7","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27-pdivcf-18","obj_delete_at_hint.0000000006","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:3","obj_delete_at_hint.0000000029","datalog.sync-status.42997773-3698-4448-bf7b-a00a5846d584","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45-pdivcf-36","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25-pdivcf-16","bilog.trim","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:8","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:5","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:5","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.5-pdivcf-5","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.1-pdivcf-44","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-48:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.5:8","obj_delete_at_hint.0000000014","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:1","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-46:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.3:8","data.full-sync.index.42997773-3698-4448-bf7b-a00a5846d584.1","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.9-pdivcf-58","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26-pdivcf-17","obj_delete_at_hint.0000000121","data_log.1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33:8","obj_delete_at_hint.0000000125","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:3","obj_delete_at_hint.0000000082","meta.full-sync.index.1","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:8","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.2-pdivcf-2","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:1","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-13:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22:1","obj_delete_at_hint.0000000077","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28:5","obj_delete_at_hint.0000000032","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:3","obj_delete_at_hint.0000000053","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:7","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-36:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45:10","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22-pdivcf-13","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21-pdivcf-12","mdlog.sync-status.shard.2","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.0.retry","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.3","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:3","obj_delete_at_hint.0000000045","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.3.retry","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.38-pdivcf-29","obj_delete_at_hint.0000000002","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:1","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.17-pdivcf-8","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.2-pdivcf-45","obj_delete_at_hint.0000000116","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:1","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-13:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22:1","obj_delete_at_hint.0000000034","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:1","obj_delete_at_hint.0000000101","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.6-pdivcf-6","mdlog.sync-status.shard.1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28:8","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","meta.log.a2b03b35-661e-4182-91b6-d60b3b65f694.3","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35-pdivcf-26","obj_delete_at_hint.0000000066","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:5","obj_delete_at_hint.0000000068","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:1","obj_delete_at_hint.0000000085","datalog.sync-status.shard.42997773-3698-4448-bf7b-a00a5846d584.1","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.3-pdivcf-3","data.full-sync.index.0378667f-02f5-4ce6-824e-e2a396c12fc3.0","meta.full-sync.index.3","obj_delete_at_hint.0000000118","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:3","data.full-sync.index.42997773-3698-4448-bf7b-a00a5846d584.0","obj_delete_at_hint.0000000036","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:5","obj_delete_at_hint.0000000103","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-38:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47:10","mdlog.sync-status.shard.3","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34:8","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","data.full-sync.index.0378667f-02f5-4ce6-824e-e2a396c12fc3.2","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","datalog.sync-status.shard.42997773-3698-4448-bf7b-a00a5846d584.3","obj_delete_at_hint.0000000039","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34:8","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-36:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45:10","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.1-pdivcf-1","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.48-pdivcf-39","obj_delete_at_hint.0000000100","obj_delete_at_hint.0000000093","datalog.sync-status.shard.42997773-3698-4448-bf7b-a00a5846d584.1.retry","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.6-pdivcf-55","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.18-pdivcf-9","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24-pdivcf-15","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.50-pdivcf-41","data_log.0","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.49-pdivcf-40","obj_delete_at_hint.0000000012","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31:8","data.full-sync.index.0378667f-02f5-4ce6-824e-e2a396c12fc3.3","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:3","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30-pdivcf-21","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-48:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.5:8","datalog.sync-status.shard.42997773-3698-4448-bf7b-a00a5846d584.2.retry","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.4-pdivcf-47","obj_delete_at_hint.0000000111","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29:8","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","meta.log.03aec9f8-1b80-4c7c-ae43-862168b0b5e6.0","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","obj_delete_at_hint.0000000072","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32:8","obj_delete_at_hint.0000000062","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31:8","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.2.retry","meta.log.30fc0b4a-5063-4d9f-a69b-5de0a471142a.3","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47-pdivcf-38","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:7","obj_delete_at_hint.0000000025","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:5","obj_delete_at_hint.0000000123","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.2","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.44-pdivcf-35","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:1","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31-pdivcf-22","obj_delete_at_hint.0000000108","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.1.retry","obj_delete_at_hint.0000000094","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.3-pdivcf-46","obj_delete_at_hint.0000000060","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-14:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23:1","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34-pdivcf-25","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32:8","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:7","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46-pdivcf-37","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-49:0378667f-02f5-4ce6-824e-e2a396c12fc3.7335.1:5","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20-pdivcf-11","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-14:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23:1","obj_delete_at_hint.0000000023","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-47:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.4:8","datalog.sync-status.shard.42997773-3698-4448-bf7b-a00a5846d584.3.retry","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:8","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32-pdivcf-23","obj_delete_at_hint.0000000107","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.5-pdivcf-48","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:8","obj_delete_at_hint.0000000046","meta.full-sync.index.2","datalog.sync-status.shard.42997773-3698-4448-bf7b-a00a5846d584.2","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.4-pdivcf-4","data_log.2","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-49:0378667f-02f5-4ce6-824e-e2a396c12fc3.7335.1:5","obj_delete_at_hint.0000000067","data.full-sync.index.0378667f-02f5-4ce6-824e-e2a396c12fc3.1","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-38:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47:10","obj_delete_at_hint.0000000086","data.full-sync.index.42997773-3698-4448-bf7b-a00a5846d584.3","datalog.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:7","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.3-pdivcf-52","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:1","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33:8","obj_delete_at_hint.0000000050","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-15:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24:1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:3","obj_delete_at_hint.0000000102","datalog.sync-status.shard.42997773-3698-4448-bf7b-a00a5846d584.0","2020-07-23-12-0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.41-pdivcf-32","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-15:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24:1","obj_delete_at_hint.0000000124","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:3","obj_delete_at_hint.0000000001","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:7","obj_delete_at_hint.0000000126","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30:8","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30:8","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-46:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.3:8","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","meta.log.03aec9f8-1b80-4c7c-ae43-862168b0b5e6.3","bucket.sync-status.42997773-3698-4448-bf7b-a00a5846d584:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:5"] 2020-07-23T12:46:48.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:48.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:48.018 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:46:48.061 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:48.061 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:48.307 INFO:teuthology.orchestra.run.smithi133.stdout:["obj_delete_at_hint.0000000078","meta.history","meta.log.03aec9f8-1b80-4c7c-ae43-862168b0b5e6.2","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-26:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.35:8","obj_delete_at_hint.0000000028","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-47:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.4:8","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:3","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:8","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:3","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28:5","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","obj_delete_at_hint.0000000096","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.0","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-37:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.46:10","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:8","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bilog.trim","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","obj_delete_at_hint.0000000014","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:1","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-24:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.33:8","obj_delete_at_hint.0000000125","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:3","obj_delete_at_hint.0000000082","meta.full-sync.index.1","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:1","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","obj_delete_at_hint.0000000053","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:7","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","mdlog.sync-status.shard.2","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.0.retry","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.3","obj_delete_at_hint.0000000045","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.3.retry","obj_delete_at_hint.0000000002","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:1","obj_delete_at_hint.0000000116","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-13:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.22:1","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-19:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.28:8","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","meta.log.a2b03b35-661e-4182-91b6-d60b3b65f694.3","obj_delete_at_hint.0000000066","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-18:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.27:5","obj_delete_at_hint.0000000068","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:1","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","data.full-sync.index.0378667f-02f5-4ce6-824e-e2a396c12fc3.0","meta.full-sync.index.3","obj_delete_at_hint.0000000118","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:3","obj_delete_at_hint.0000000036","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-17:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.26:5","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","data.full-sync.index.0378667f-02f5-4ce6-824e-e2a396c12fc3.2","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","obj_delete_at_hint.0000000039","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-25:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.34:8","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-36:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.45:10","obj_delete_at_hint.0000000100","obj_delete_at_hint.0000000093","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","obj_delete_at_hint.0000000012","data.full-sync.index.0378667f-02f5-4ce6-824e-e2a396c12fc3.3","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-48:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.5:8","obj_delete_at_hint.0000000111","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-20:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.29:8","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","meta.log.03aec9f8-1b80-4c7c-ae43-862168b0b5e6.0","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","obj_delete_at_hint.0000000072","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-23:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.32:8","obj_delete_at_hint.0000000062","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-22:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.31:8","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.2.retry","meta.log.30fc0b4a-5063-4d9f-a69b-5de0a471142a.3","obj_delete_at_hint.0000000025","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:5","obj_delete_at_hint.0000000123","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.2","obj_delete_at_hint.0000000108","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.1.retry","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-14:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.23:1","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:7","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-11:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.20:8","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","meta.full-sync.index.2","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-49:0378667f-02f5-4ce6-824e-e2a396c12fc3.7335.1:5","obj_delete_at_hint.0000000067","data.full-sync.index.0378667f-02f5-4ce6-824e-e2a396c12fc3.1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-38:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.47:10","obj_delete_at_hint.0000000086","datalog.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-10:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.19:1","obj_delete_at_hint.0000000050","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-15:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.24:1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:3","obj_delete_at_hint.0000000102","obj_delete_at_hint.0000000124","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-16:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.25:3","obj_delete_at_hint.0000000001","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-12:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.21:7","obj_delete_at_hint.0000000126","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-21:0378667f-02f5-4ce6-824e-e2a396c12fc3.4476.30:8","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","datalog.sync-status.shard.0378667f-02f5-4ce6-824e-e2a396c12fc3.1","bucket.sync-status.0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-46:0378667f-02f5-4ce6-824e-e2a396c12fc3.6993.3:8","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","meta.log.03aec9f8-1b80-4c7c-ae43-862168b0b5e6.3"] 2020-07-23T12:46:48.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:48.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:48.438 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:46:48.478 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:48.479 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:48.741 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595508406.893877_230.3","last_update":"2020-07-23T12:46:46.893877Z"},{"marker":"1_1595508406.893877_619.2","last_update":"2020-07-23T12:46:46.893877Z"},{"marker":"1_1595508406.893877_433.3","last_update":"2020-07-23T12:46:46.893877Z"},{"marker":"1_1595508406.893877_357.3","last_update":"2020-07-23T12:46:46.893877Z"}] 2020-07-23T12:46:48.742 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-23T12:46:48.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:48.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:48.745 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:46:48.798 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:48.799 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:49.038 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":988729214735988972},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.265297_92.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.265297Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507675.998941_262.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:35.998941Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508053.128510_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:53.128510Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508263.920641_291.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:44:23.920641Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:46:49.039 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1595508406.893877_230.3 target=1_1595507891.265297_92.1, shard=1 master=1_1595508406.893877_619.2 target=1_1595507675.998941_262.2, shard=2 master=1_1595508406.893877_433.3 target=1_1595508053.128510_296.1, shard=3 master=1_1595508406.893877_357.3 target=1_1595508263.920641_291.1 2020-07-23T12:46:54.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:54.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:54.043 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:46:54.085 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:54.085 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:54.330 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":988729214735988972},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.265297_92.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.265297Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507675.998941_262.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:35.998941Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508053.128510_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:53.128510Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508263.920641_291.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:44:23.920641Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:46:54.331 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1595508406.893877_230.3 target=1_1595507891.265297_92.1, shard=1 master=1_1595508406.893877_619.2 target=1_1595507675.998941_262.2, shard=2 master=1_1595508406.893877_433.3 target=1_1595508053.128510_296.1, shard=3 master=1_1595508406.893877_357.3 target=1_1595508263.920641_291.1 2020-07-23T12:46:59.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:59.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:46:59.343 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:46:59.385 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:46:59.386 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:46:59.640 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":988729214735988972},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.265297_92.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.265297Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507675.998941_262.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:35.998941Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508053.128510_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:53.128510Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508263.920641_291.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:44:23.920641Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:46:59.640 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1595508406.893877_230.3 target=1_1595507891.265297_92.1, shard=1 master=1_1595508406.893877_619.2 target=1_1595507675.998941_262.2, shard=2 master=1_1595508406.893877_433.3 target=1_1595508053.128510_296.1, shard=3 master=1_1595508406.893877_357.3 target=1_1595508263.920641_291.1 2020-07-23T12:47:04.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:04.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:04.644 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:04.687 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:04.687 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:04.942 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":988729214735988972},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.265297_92.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.265297Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507675.998941_262.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:35.998941Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508053.128510_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:53.128510Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508263.920641_291.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:44:23.920641Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:47:04.942 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1595508406.893877_230.3 target=1_1595507891.265297_92.1, shard=1 master=1_1595508406.893877_619.2 target=1_1595507675.998941_262.2, shard=2 master=1_1595508406.893877_433.3 target=1_1595508053.128510_296.1, shard=3 master=1_1595508406.893877_357.3 target=1_1595508263.920641_291.1 2020-07-23T12:47:09.883 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:09.880+0000 7fd099feb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:09.883 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:09.880+0000 7fd09c7f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:09.883 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:09.880+0000 7fd0a0ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:09.884 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:09.880+0000 7fd098fe9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:09.884 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:09.880+0000 7fd09fff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:09.885 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:09.880+0000 7fd09b7ee700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:09.885 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:09.880+0000 7fd0ab7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:09.885 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:09.880+0000 7fd09afed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:09.885 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:09.880+0000 7fd0c0d54700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:09.885 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:09.880+0000 7fd0997ea700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:09.886 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:09.880+0000 7fd0aaffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:09.886 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:09.880+0000 7fd0a1ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:09.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:09.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:09.946 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:10.001 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:10.001 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:10.250 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":988729214735988972},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.265297_92.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.265297Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507675.998941_262.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:35.998941Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508053.128510_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:53.128510Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508263.920641_291.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:44:23.920641Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:47:10.251 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1595508406.893877_230.3 target=1_1595507891.265297_92.1, shard=1 master=1_1595508406.893877_619.2 target=1_1595507675.998941_262.2, shard=2 master=1_1595508406.893877_433.3 target=1_1595508053.128510_296.1, shard=3 master=1_1595508406.893877_357.3 target=1_1595508263.920641_291.1 2020-07-23T12:47:11.578 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f4745ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.579 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f47417f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.579 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f473ffef700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.579 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f473efed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.579 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f4744ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.580 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f47467fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.580 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f474e7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.580 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f4743ff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.581 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f47437f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.581 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f473f7ee700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.581 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f4741ff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.582 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f47427f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.582 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f47457fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.582 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f4759ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.582 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f473dfeb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.582 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f475a7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.583 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f47447f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.583 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.573+0000 7f47477fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.588 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.581+0000 7f47427f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.588 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.581+0000 7f4759ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.589 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.581+0000 7f47457fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.589 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.581+0000 7f475a7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.590 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.581+0000 7f474dffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.598 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.593+0000 7f47407f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.599 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.593+0000 7f4745ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.599 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.593+0000 7f473efed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.599 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.593+0000 7f473e7ec700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:11.599 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:11.593+0000 7f4746ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:15.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:15.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:15.267 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:15.317 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:15.318 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:15.565 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":988729214735988972},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.265297_92.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.265297Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507675.998941_262.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:35.998941Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508053.128510_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:53.128510Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508263.920641_291.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:44:23.920641Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:47:15.565 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1595508406.893877_230.3 target=1_1595507891.265297_92.1, shard=1 master=1_1595508406.893877_619.2 target=1_1595507675.998941_262.2, shard=2 master=1_1595508406.893877_433.3 target=1_1595508053.128510_296.1, shard=3 master=1_1595508406.893877_357.3 target=1_1595508263.920641_291.1 2020-07-23T12:47:20.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:20.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:20.577 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:20.623 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:20.624 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:20.861 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":988729214735988972},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.265297_92.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.265297Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507675.998941_262.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:35.998941Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508053.128510_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:53.128510Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508263.920641_291.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:44:23.920641Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:47:20.862 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1595508406.893877_230.3 target=1_1595507891.265297_92.1, shard=1 master=1_1595508406.893877_619.2 target=1_1595507675.998941_262.2, shard=2 master=1_1595508406.893877_433.3 target=1_1595508053.128510_296.1, shard=3 master=1_1595508406.893877_357.3 target=1_1595508263.920641_291.1 2020-07-23T12:47:25.865 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:25.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:25.867 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:25.914 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:25.915 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:26.183 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":988729214735988972},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.265297_92.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.265297Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507675.998941_262.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:35.998941Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508053.128510_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:53.128510Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508263.920641_291.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:44:23.920641Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:47:26.183 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1595508406.893877_230.3 target=1_1595507891.265297_92.1, shard=1 master=1_1595508406.893877_619.2 target=1_1595507675.998941_262.2, shard=2 master=1_1595508406.893877_433.3 target=1_1595508053.128510_296.1, shard=3 master=1_1595508406.893877_357.3 target=1_1595508263.920641_291.1 2020-07-23T12:47:31.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:31.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:31.187 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:31.230 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:31.230 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:31.488 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":988729214735988972},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.265297_92.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.265297Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507675.998941_262.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:35.998941Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508053.128510_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:53.128510Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508263.920641_291.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:44:23.920641Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:47:31.489 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1595508406.893877_230.3 target=1_1595507891.265297_92.1, shard=1 master=1_1595508406.893877_619.2 target=1_1595507675.998941_262.2, shard=2 master=1_1595508406.893877_433.3 target=1_1595508053.128510_296.1, shard=3 master=1_1595508406.893877_357.3 target=1_1595508263.920641_291.1 2020-07-23T12:47:36.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:36.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:36.494 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:36.538 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:36.539 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:36.796 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":988729214735988972},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.265297_92.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.265297Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507675.998941_262.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:35.998941Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508053.128510_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:53.128510Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508263.920641_291.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:44:23.920641Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:47:36.797 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1595508406.893877_230.3 target=1_1595507891.265297_92.1, shard=1 master=1_1595508406.893877_619.2 target=1_1595507675.998941_262.2, shard=2 master=1_1595508406.893877_433.3 target=1_1595508053.128510_296.1, shard=3 master=1_1595508406.893877_357.3 target=1_1595508263.920641_291.1 2020-07-23T12:47:41.402 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.397+0000 7f019cff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.403 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.397+0000 7f019d7f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.412 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.397+0000 7f019c7f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.412 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.397+0000 7f019a7ec700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.412 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.397+0000 7f019afed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.413 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.397+0000 7f01abfff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.413 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.401+0000 7f01a3fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.414 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.401+0000 7f01aa7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.451 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.437+0000 7f01a0ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.451 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.437+0000 7f019fff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.451 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.441+0000 7f019bfef700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.452 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.441+0000 7f019f7f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.452 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.441+0000 7f01a37fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.452 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.441+0000 7f01a17fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.452 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.441+0000 7f01997ea700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.452 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.441+0000 7f019afed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.462 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.449+0000 7f01a17fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.463 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.449+0000 7f01997ea700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.463 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:47:41.449+0000 7f019afed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:41.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:41.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:41.800 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:41.844 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:41.844 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:42.107 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":988729214735988972},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508406.893877_230.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:46:46.893877Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508406.893877_619.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:46:46.893877Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508406.893877_433.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:46:46.893877Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508406.893877_357.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:46:46.893877Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:47:42.108 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-23T12:47:42.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:42.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:42.110 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:42.150 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:42.151 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:42.429 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595508406.893877_230.3","last_update":"2020-07-23T12:46:46.893877Z"},{"marker":"1_1595508406.893877_619.2","last_update":"2020-07-23T12:46:46.893877Z"},{"marker":"1_1595508406.893877_433.3","last_update":"2020-07-23T12:46:46.893877Z"},{"marker":"1_1595508406.893877_357.3","last_update":"2020-07-23T12:46:46.893877Z"}] 2020-07-23T12:47:42.430 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-23T12:47:42.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:42.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:42.433 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:42.512 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:42.512 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:42.785 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":5372564157323408779},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508406.893877_230.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:46:46.893877Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508406.893877_619.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:46:46.893877Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508406.893877_433.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:46:46.893877Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508406.893877_357.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:46:46.893877Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:47:42.785 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-23T12:47:42.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:42.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:42.787 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:42.864 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:42.865 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:43.127 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595508406.893877_230.3","last_update":"2020-07-23T12:46:46.893877Z"},{"marker":"1_1595508406.893877_619.2","last_update":"2020-07-23T12:46:46.893877Z"},{"marker":"1_1595508406.893877_433.3","last_update":"2020-07-23T12:46:46.893877Z"},{"marker":"1_1595508406.893877_357.3","last_update":"2020-07-23T12:46:46.893877Z"}] 2020-07-23T12:47:43.127 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-23T12:47:43.128 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:43.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:43.129 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:43.182 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:43.183 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:43.455 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1496412237659215512},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508406.893877_230.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:46:46.893877Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508406.893877_619.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:46:46.893877Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508406.893877_433.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:46:46.893877Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508406.893877_357.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:46:46.893877Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:47:43.456 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-23T12:47:43.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:43.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:43.458 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:43.535 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:43.536 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:43.786 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1595507891.299750_60.1","last_update":"2020-07-23T12:38:11.299750Z"},{"marker":"1_1595507676.941349_266.3","last_update":"2020-07-23T12:34:36.941349Z"},{"marker":"1_1595508059.093643_244.1","last_update":"2020-07-23T12:40:59.093643Z"},{"marker":"1_1595508309.223978_306.1","last_update":"2020-07-23T12:45:09.223978Z"}] 2020-07-23T12:47:43.786 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-23T12:47:43.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:43.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:43.788 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:43.832 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:43.832 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:44.095 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14254013310941169947},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.299750_60.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.299750Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507676.941349_266.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:36.941349Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508059.093643_244.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:59.093643Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508309.223978_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:45:09.223978Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:47:44.096 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-23T12:47:44.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:44.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:44.098 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:44.159 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:44.159 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:44.391 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1595507891.299750_60.1","last_update":"2020-07-23T12:38:11.299750Z"},{"marker":"1_1595507676.941349_266.3","last_update":"2020-07-23T12:34:36.941349Z"},{"marker":"1_1595508059.093643_244.1","last_update":"2020-07-23T12:40:59.093643Z"},{"marker":"1_1595508309.223978_306.1","last_update":"2020-07-23T12:45:09.223978Z"}] 2020-07-23T12:47:44.392 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-23T12:47:44.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:44.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:44.394 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:44.441 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:44.441 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:44.702 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11610391562858473267},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595507891.299750_60.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:38:11.299750Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595507676.941349_266.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:34:36.941349Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508059.093643_244.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:40:59.093643Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508309.223978_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:45:09.223978Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:47:44.703 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-23T12:47:44.704 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:44.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:44.705 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:44.784 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:44.784 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:45.034 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595508434.304781_190.1","last_update":"2020-07-23T12:47:14.304781Z"},{"marker":"1_1595508434.287949_514.1","last_update":"2020-07-23T12:47:14.287949Z"},{"marker":"1_1595508434.294997_309.1","last_update":"2020-07-23T12:47:14.294997Z"},{"marker":"1_1595508434.300562_269.1","last_update":"2020-07-23T12:47:14.300562Z"}] 2020-07-23T12:47:45.035 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-23T12:47:45.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:45.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:45.037 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:45.124 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:45.125 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:45.385 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14010726936006743351},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508434.304781_190.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:47:14.304781Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508434.287949_514.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:47:14.287949Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508434.294997_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:47:14.294997Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508434.300562_269.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:47:14.300562Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:47:45.385 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-23T12:47:45.386 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:45.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:45.387 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:45.472 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:45.473 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:45.731 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595508434.304781_190.1","last_update":"2020-07-23T12:47:14.304781Z"},{"marker":"1_1595508434.287949_514.1","last_update":"2020-07-23T12:47:14.287949Z"},{"marker":"1_1595508434.294997_309.1","last_update":"2020-07-23T12:47:14.294997Z"},{"marker":"1_1595508434.300562_269.1","last_update":"2020-07-23T12:47:14.300562Z"}] 2020-07-23T12:47:45.732 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-23T12:47:45.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:45.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:45.733 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:45.777 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:45.777 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:46.042 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3438718294491381619},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508434.304781_190.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:47:14.304781Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508434.287949_514.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:47:14.287949Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508434.294997_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:47:14.294997Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508434.300562_269.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:47:14.300562Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:47:46.043 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-23T12:47:46.045 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... ok 2020-07-23T12:47:46.046 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-59 2020-07-23T12:47:46.096 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-60 2020-07-23T12:47:46.171 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-61 2020-07-23T12:47:46.602 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:46.595+0000 7fd09afed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:46.603 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:46.595+0000 7fd0a3fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:46.603 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:46.599+0000 7fd09dff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:46.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:46.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:46.876 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:46.918 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:46.918 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:47.182 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.56.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.54.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.30.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.44.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:47:47.183 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-59 2020-07-23T12:47:47.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:47.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:47.185 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:47.239 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:47.239 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:47.261 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:47.255+0000 7fd0a17fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:47.262 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:47.257+0000 7f47467fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:47.266 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:47.261+0000 7f4742ff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:47.267 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:47.261+0000 7f473e7ec700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:47.267 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:47.263+0000 7fd0a27fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:47.268 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:47:47.263+0000 7fd0a8ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:47.269 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:47.261+0000 7f4744ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:47.269 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:47:47.261+0000 7f47447f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:47:47.511 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:47:47.511 WARNING:rgw_multi.tests:bucket pdivcf-59 zone test-zone2 behind zone test-zone1: shard=2 master=00000000001.54.5 target=, shard=7 master=00000000001.44.5 target= 2020-07-23T12:47:52.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:52.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:52.515 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:52.559 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:52.560 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:52.833 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:47:52.834 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-59 2020-07-23T12:47:52.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:52.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:52.838 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:52.891 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:52.893 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:53.147 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.56.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.54.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.30.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.44.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:47:53.148 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-59 2020-07-23T12:47:53.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:53.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:53.150 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:53.231 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:53.232 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:53.513 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:47:53.514 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-59 2020-07-23T12:47:53.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:53.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:53.517 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:53.612 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:53.613 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:53.866 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.56.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.54.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.30.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.44.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:47:53.867 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-59 2020-07-23T12:47:53.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:53.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:53.869 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:53.916 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:53.916 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:54.193 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.56.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.54.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.30.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.44.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:47:54.194 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-59 2020-07-23T12:47:54.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:54.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:54.195 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:54.279 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:54.279 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:54.558 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000001.83.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.54.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.36.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.44.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:47:54.559 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-59 2020-07-23T12:47:54.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:54.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:54.561 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:54.618 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:54.619 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:54.912 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.36.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:47:54.912 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-59 2020-07-23T12:47:54.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:54.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:54.914 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:54.964 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:54.964 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:55.220 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000001.83.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.54.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.36.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.44.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:47:55.220 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-59 2020-07-23T12:47:55.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:55.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:55.222 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:55.267 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:55.267 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:55.548 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.36.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:47:55.549 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-59 2020-07-23T12:47:55.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:55.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:55.551 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:55.630 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:55.631 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:55.898 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.69.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.64.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.37.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.42.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:47:55.899 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-59 2020-07-23T12:47:55.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:55.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:55.902 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:55.985 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:55.986 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:56.263 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:47:56.264 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-59 2020-07-23T12:47:56.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:56.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:56.266 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:56.344 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:56.345 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:56.602 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.69.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.64.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.37.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.42.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:47:56.602 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-59 2020-07-23T12:47:56.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:56.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:47:56.604 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:47:56.648 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:47:56.649 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:47:56.923 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:47:56.924 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-59 2020-07-23T12:47:56.924 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-59 zones={test-zone2, test-zone1} 2020-07-23T12:47:57.794 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:47:57.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:47:57.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:47:57.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:47:57.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:47:57.812 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:47:57.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:47:57.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:47:57.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:47:57.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:47:57.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:47:57.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:47:58.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:47:58.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:47:58.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:47:58.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:47:58.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:47:58.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:47:58.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:47:58.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:47:58.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:47:58.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:47:59.052 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-59 zones={test-zone2, test-zone1} 2020-07-23T12:47:59.053 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-59 zones={test-zone3, test-zone1} 2020-07-23T12:47:59.906 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:47:59.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:47:59.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:47:59.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:47:59.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:47:59.915 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:47:59.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:47:59.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:47:59.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:47:59.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:47:59.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:47:59.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:00.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:00.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:00.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:00.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:00.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:00.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:00.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:00.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:00.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:00.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:01.050 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-59 zones={test-zone3, test-zone1} 2020-07-23T12:48:01.051 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-59 zones={test-zone3, test-zone2} 2020-07-23T12:48:01.070 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:48:01.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:01.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:01.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:01.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:01.079 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:48:01.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:01.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:01.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:01.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:01.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:01.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:01.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:01.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:01.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:01.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:01.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:01.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:01.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:01.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:02.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:02.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:02.229 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-59 zones={test-zone3, test-zone2} 2020-07-23T12:48:02.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:02.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:02.231 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:02.270 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:02.271 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:02.553 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.37.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.58.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.44.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.55.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:48:02.554 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-60 2020-07-23T12:48:02.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:02.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:02.556 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:02.602 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:02.602 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:02.884 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:48:02.885 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-60 2020-07-23T12:48:02.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:02.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:02.887 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:02.929 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:02.929 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:03.179 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.37.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.58.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.44.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.55.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:48:03.180 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-60 2020-07-23T12:48:03.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:03.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:03.182 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:03.265 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:03.265 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:03.564 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:48:03.565 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-60 2020-07-23T12:48:03.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:03.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:03.568 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:03.650 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:03.650 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:03.916 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.37.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.58.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.44.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.55.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:48:03.917 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-60 2020-07-23T12:48:03.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:03.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:03.919 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:03.963 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:03.963 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:04.241 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.37.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.58.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.44.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.55.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:48:04.242 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-60 2020-07-23T12:48:04.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:04.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:04.244 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:04.330 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:04.331 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:04.597 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000001.34.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.79.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.46.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.39.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:48:04.598 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-60 2020-07-23T12:48:04.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:04.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:04.600 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:04.649 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:04.649 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:04.925 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.34.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.79.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:48:04.926 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-60 2020-07-23T12:48:04.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:04.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:04.928 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:04.968 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:04.968 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:05.222 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000001.34.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.79.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.46.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.39.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:48:05.222 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-60 2020-07-23T12:48:05.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:05.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:05.224 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:05.269 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:05.270 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:05.548 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.34.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.79.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:48:05.549 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-60 2020-07-23T12:48:05.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:05.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:05.552 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:05.632 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:05.632 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:05.893 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.28.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.63.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.42.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.41.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:48:05.894 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-60 2020-07-23T12:48:05.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:05.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:05.897 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:05.978 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:05.979 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:06.237 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:48:06.237 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-60 2020-07-23T12:48:06.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:06.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:06.241 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:06.284 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:06.284 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:06.557 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.28.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.63.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.42.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.41.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:48:06.558 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-60 2020-07-23T12:48:06.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:06.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:06.560 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:06.607 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:06.608 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:06.888 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:48:06.889 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-60 2020-07-23T12:48:06.890 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-60 zones={test-zone2, test-zone1} 2020-07-23T12:48:08.135 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:48:08.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:08.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:08.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:08.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:08.162 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:48:08.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:08.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:08.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:08.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:08.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:08.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:08.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:08.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:08.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:08.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:08.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:08.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:08.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:09.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:09.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:09.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:09.368 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-60 zones={test-zone2, test-zone1} 2020-07-23T12:48:09.369 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-60 zones={test-zone3, test-zone1} 2020-07-23T12:48:10.137 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:48:10.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:10.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:10.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:10.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:10.144 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:48:10.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:10.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:10.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:10.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:10.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:10.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:10.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:10.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:10.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:10.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:10.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:10.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:10.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:11.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:11.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:11.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:11.341 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-60 zones={test-zone3, test-zone1} 2020-07-23T12:48:11.342 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-60 zones={test-zone3, test-zone2} 2020-07-23T12:48:11.814 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:48:11.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:11.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:11.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:11.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:11.825 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:48:11.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:11.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:11.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:11.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:11.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:12.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:12.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:12.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:12.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:12.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:12.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:12.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:12.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:12.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:12.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:13.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:13.160 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-60 zones={test-zone3, test-zone2} 2020-07-23T12:48:13.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:13.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:13.169 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:13.227 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:13.228 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:13.494 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.60.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.50.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.39.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.77.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:48:13.495 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-61 2020-07-23T12:48:13.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:13.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:13.496 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:13.542 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:13.542 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:13.813 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.77.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:48:13.814 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-61 2020-07-23T12:48:13.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:13.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:13.816 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:13.863 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:13.864 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:14.118 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.60.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.50.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.39.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.77.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:48:14.119 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-61 2020-07-23T12:48:14.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:14.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:14.121 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:14.206 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:14.207 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:14.491 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.77.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:48:14.492 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-61 2020-07-23T12:48:14.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:14.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:14.494 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:14.579 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:14.580 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:14.840 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.60.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.50.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.39.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.77.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:48:14.843 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-61 2020-07-23T12:48:14.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:14.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:14.856 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:14.908 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:14.908 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:15.183 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.60.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.50.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.39.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.77.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:48:15.185 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-61 2020-07-23T12:48:15.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:15.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:15.186 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:15.268 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:15.269 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:15.526 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000001.60.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.50.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.42.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.69.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:48:15.526 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-61 2020-07-23T12:48:15.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:15.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:15.528 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:15.573 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:15.574 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:15.850 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:48:15.851 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-61 2020-07-23T12:48:15.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:15.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:15.853 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:15.898 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:15.898 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:16.158 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000001.60.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.50.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.42.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.69.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:48:16.159 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-61 2020-07-23T12:48:16.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:16.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:16.170 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:16.221 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:16.222 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:16.482 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:48:16.483 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-61 2020-07-23T12:48:16.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:16.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:16.485 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:16.566 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:16.567 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:16.827 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.48.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.45.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.61.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:48:16.827 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-61 2020-07-23T12:48:16.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:16.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:16.829 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:16.912 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:16.912 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:17.170 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:48:17.171 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-61 2020-07-23T12:48:17.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:17.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:17.173 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:17.266 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:17.266 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:17.502 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.48.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.45.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.61.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:48:17.502 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-61 2020-07-23T12:48:17.503 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:17.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:17.505 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:17.554 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:17.555 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:17.821 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:48:17.822 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-61 2020-07-23T12:48:17.822 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-61 zones={test-zone2, test-zone1} 2020-07-23T12:48:19.154 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:48:19.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:19.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:19.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:19.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:19.165 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:48:19.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:19.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:19.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:19.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:19.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:19.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:19.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:19.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:19.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:19.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:19.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:19.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:20.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:20.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:20.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:20.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:20.416 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-61 zones={test-zone2, test-zone1} 2020-07-23T12:48:20.417 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-61 zones={test-zone3, test-zone1} 2020-07-23T12:48:21.189 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:48:21.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:21.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:21.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:21.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:21.198 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:48:21.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:21.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:21.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:21.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:21.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:21.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:21.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:21.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:21.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:21.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:21.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:21.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:21.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:22.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:22.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:22.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:22.384 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-61 zones={test-zone3, test-zone1} 2020-07-23T12:48:22.384 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-61 zones={test-zone3, test-zone2} 2020-07-23T12:48:23.218 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:48:23.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:23.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:23.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:23.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:23.232 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:48:23.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:23.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:23.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:23.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:23.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:23.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:23.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:23.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:23.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:23.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:23.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:23.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:24.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:24.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:24.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:24.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:24.465 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-61 zones={test-zone3, test-zone2} 2020-07-23T12:48:24.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'pdivcf-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:48:24.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'pdivcf-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:48:24.468 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket pdivcf-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:48:24.512 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:24.512 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:24.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'pdivcf-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:48:24.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'pdivcf-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:48:24.888 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket pdivcf-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:48:24.980 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:24.980 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:25.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'pdivcf-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:48:25.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'pdivcf-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:48:25.355 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket pdivcf-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:48:25.442 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:25.443 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:25.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'pdivcf-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:48:25.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'pdivcf-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:48:25.804 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket pdivcf-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:48:25.897 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:25.903 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:26.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'pdivcf-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:48:26.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'pdivcf-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:48:26.262 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket pdivcf-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:48:26.342 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:26.342 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:26.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'pdivcf-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:48:26.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'pdivcf-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:48:26.749 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket pdivcf-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:48:26.830 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:26.830 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:29.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:29.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:29.488 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:29.528 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:29.529 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:29.773 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.72.5"},{"key":1,"val":"00000000004.48.1"},{"key":2,"val":"00000000006.64.5"},{"key":3,"val":"00000000004.49.1"},{"key":4,"val":"00000000006.48.5"},{"key":5,"val":"00000000004.54.1"},{"key":6,"val":"00000000004.41.1"},{"key":7,"val":"00000000006.50.5"},{"key":8,"val":"00000000004.42.1"},{"key":9,"val":"00000000004.91.1"},{"key":10,"val":"00000000004.43.1"}]} 2020-07-23T12:48:29.774 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-59 2020-07-23T12:48:29.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:29.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:29.776 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:29.820 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:29.820 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:30.090 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.72.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.64.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.48.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.50.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}}] 2020-07-23T12:48:30.091 WARNING:rgw_multi.tests:bucket pdivcf-59 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.48.1 target=, shard=5 master=00000000004.54.1 target=, shard=9 master=00000000004.91.1 target= 2020-07-23T12:48:35.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:35.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:35.094 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:35.138 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:35.139 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:35.402 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.72.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.64.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.48.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.50.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}}] 2020-07-23T12:48:35.403 WARNING:rgw_multi.tests:bucket pdivcf-59 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.48.1 target=, shard=5 master=00000000004.54.1 target=, shard=9 master=00000000004.91.1 target= 2020-07-23T12:48:40.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:40.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:40.406 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:40.451 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:40.452 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:40.732 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.72.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.64.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.48.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.50.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}}] 2020-07-23T12:48:40.734 WARNING:rgw_multi.tests:bucket pdivcf-59 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.48.1 target=, shard=5 master=00000000004.54.1 target=, shard=9 master=00000000004.91.1 target= 2020-07-23T12:48:45.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:45.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:45.739 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:45.786 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:45.786 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:46.055 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.72.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.64.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.48.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.50.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.91.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}}] 2020-07-23T12:48:46.056 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-59 2020-07-23T12:48:46.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:46.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:46.059 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:46.105 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:46.106 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:46.381 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.72.5"},{"key":1,"val":"00000000004.48.1"},{"key":2,"val":"00000000006.64.5"},{"key":3,"val":"00000000004.49.1"},{"key":4,"val":"00000000006.48.5"},{"key":5,"val":"00000000004.54.1"},{"key":6,"val":"00000000004.41.1"},{"key":7,"val":"00000000006.50.5"},{"key":8,"val":"00000000004.42.1"},{"key":9,"val":"00000000004.91.1"},{"key":10,"val":"00000000004.43.1"}]} 2020-07-23T12:48:46.382 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-59 2020-07-23T12:48:46.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:46.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:46.384 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:46.474 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:46.475 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:46.726 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.72.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.64.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.48.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.50.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.91.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}}] 2020-07-23T12:48:46.727 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-59 2020-07-23T12:48:46.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:46.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:46.729 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:46.811 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:46.811 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:47.079 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.72.5"},{"key":1,"val":"00000000004.48.1"},{"key":2,"val":"00000000006.64.5"},{"key":3,"val":"00000000004.49.1"},{"key":4,"val":"00000000006.48.5"},{"key":5,"val":"00000000004.54.1"},{"key":6,"val":"00000000004.41.1"},{"key":7,"val":"00000000006.50.5"},{"key":8,"val":"00000000004.42.1"},{"key":9,"val":"00000000004.91.1"},{"key":10,"val":"00000000004.43.1"}]} 2020-07-23T12:48:47.079 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-59 2020-07-23T12:48:47.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:47.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:47.081 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:47.130 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:47.130 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:47.401 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.72.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.64.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.48.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.50.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.91.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}}] 2020-07-23T12:48:47.402 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-59 2020-07-23T12:48:47.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:47.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:47.416 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:47.458 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:47.459 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:47.713 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000004.95.5"},{"key":1,"val":"00000000002.48.1"},{"key":2,"val":"00000000004.62.5"},{"key":3,"val":"00000000002.47.1"},{"key":4,"val":"00000000004.52.5"},{"key":5,"val":"00000000002.86.1"},{"key":6,"val":"00000000002.44.1"},{"key":7,"val":"00000000004.50.5"},{"key":8,"val":"00000000002.45.1"},{"key":9,"val":"00000000002.96.1"},{"key":10,"val":"00000000002.44.1"}]} 2020-07-23T12:48:47.714 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-59 2020-07-23T12:48:47.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:47.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:47.717 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:47.767 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:47.768 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:48.038 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.95.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.48.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.62.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.52.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.44.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.50.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.96.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.44.1"}}}] 2020-07-23T12:48:48.039 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-59 2020-07-23T12:48:48.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:48.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:48.041 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:48.082 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:48.083 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:48.336 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000004.95.5"},{"key":1,"val":"00000000002.48.1"},{"key":2,"val":"00000000004.62.5"},{"key":3,"val":"00000000002.47.1"},{"key":4,"val":"00000000004.52.5"},{"key":5,"val":"00000000002.86.1"},{"key":6,"val":"00000000002.44.1"},{"key":7,"val":"00000000004.50.5"},{"key":8,"val":"00000000002.45.1"},{"key":9,"val":"00000000002.96.1"},{"key":10,"val":"00000000002.44.1"}]} 2020-07-23T12:48:48.337 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-59 2020-07-23T12:48:48.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:48.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:48.339 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:48.383 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:48.384 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:48.675 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.95.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.48.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.62.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.52.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.44.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.50.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.96.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.44.1"}}}] 2020-07-23T12:48:48.676 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-59 2020-07-23T12:48:48.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:48.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:48.678 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:48.758 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:48.759 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:49.004 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000002.76.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.69.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.46.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.46.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:48:49.005 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-59 2020-07-23T12:48:49.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:49.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:49.007 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:49.093 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:49.094 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:49.392 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.76.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.69.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.46.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.46.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:48:49.393 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-59 2020-07-23T12:48:49.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:49.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:49.396 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:49.440 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:49.441 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:49.692 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000002.76.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.69.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.46.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.46.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:48:49.693 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-59 2020-07-23T12:48:49.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:49.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:49.695 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:49.738 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:49.739 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:50.011 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.76.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.69.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.46.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.46.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:48:50.012 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-59 2020-07-23T12:48:50.013 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-59 zones={test-zone2, test-zone1} 2020-07-23T12:48:51.623 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:48:51.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:51.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:51.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:51.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:51.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:48:51.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:48:51.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:48:51.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:48:51.650 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:48:51.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:51.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:51.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:51.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:51.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:48:51.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:48:51.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:48:51.669 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:48:51.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:51.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:51.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:51.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:52.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:52.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:52.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:52.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:52.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:52.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:52.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:52.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:52.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:48:52.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:48:53.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:48:53.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:48:53.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:48:53.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:48:53.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:48:53.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:48:53.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:48:53.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:48:53.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:48:53.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:48:54.013 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-59 zones={test-zone2, test-zone1} 2020-07-23T12:48:54.013 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-59 zones={test-zone3, test-zone1} 2020-07-23T12:48:54.106 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:48:54.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:54.113 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:54.113 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:54.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:54.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:48:54.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:48:54.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:48:54.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:48:54.116 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:48:54.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:54.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:54.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:54.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:54.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:48:54.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:48:54.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:48:54.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:48:54.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:54.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:54.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:54.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:54.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:54.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:54.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:54.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:54.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:55.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:55.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:55.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:55.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:48:55.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:48:55.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:48:55.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:48:55.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:48:55.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:48:55.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:48:55.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:48:56.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:48:56.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:48:56.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:48:56.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:48:56.590 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-59 zones={test-zone3, test-zone1} 2020-07-23T12:48:56.590 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-59 zones={test-zone3, test-zone2} 2020-07-23T12:48:57.015 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:48:57.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:57.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:57.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:57.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:57.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:48:57.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:48:57.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:48:57.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:48:57.029 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:48:57.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:48:57.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:48:57.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:48:57.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:48:57.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:48:57.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:48:57.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:48:57.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:48:57.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:57.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:57.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:48:57.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:57.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:57.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:48:57.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:57.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:57.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:48:57.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:58.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:58.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:48:58.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:48:58.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:48:58.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:48:58.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:48:58.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:48:58.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:48:58.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:48:58.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:48:59.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:48:59.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:48:59.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:48:59.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:48:59.409 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-59 zones={test-zone3, test-zone2} 2020-07-23T12:48:59.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:59.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:59.411 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:59.458 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:59.459 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:48:59.765 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.43.5"},{"key":1,"val":"00000000004.58.1"},{"key":2,"val":"00000000006.74.5"},{"key":3,"val":"00000000004.54.1"},{"key":4,"val":"00000000006.50.5"},{"key":5,"val":"00000000004.64.1"},{"key":6,"val":"00000000004.45.1"},{"key":7,"val":"00000000006.61.5"},{"key":8,"val":"00000000004.81.1"},{"key":9,"val":"00000000004.95.1"},{"key":10,"val":"00000000004.71.1"}]} 2020-07-23T12:48:59.766 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-60 2020-07-23T12:48:59.767 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:59.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:48:59.769 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:48:59.814 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:48:59.815 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:00.092 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.43.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.74.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.50.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.61.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.81.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.95.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}}] 2020-07-23T12:49:00.093 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-60 2020-07-23T12:49:00.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:00.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:00.095 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:00.136 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:00.137 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:00.409 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.43.5"},{"key":1,"val":"00000000004.58.1"},{"key":2,"val":"00000000006.74.5"},{"key":3,"val":"00000000004.54.1"},{"key":4,"val":"00000000006.50.5"},{"key":5,"val":"00000000004.64.1"},{"key":6,"val":"00000000004.45.1"},{"key":7,"val":"00000000006.61.5"},{"key":8,"val":"00000000004.81.1"},{"key":9,"val":"00000000004.95.1"},{"key":10,"val":"00000000004.71.1"}]} 2020-07-23T12:49:00.410 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-60 2020-07-23T12:49:00.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:00.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:00.412 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:00.495 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:00.496 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:00.772 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.43.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.74.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.50.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.61.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.81.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.95.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}}] 2020-07-23T12:49:00.773 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-60 2020-07-23T12:49:00.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:00.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:00.775 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:00.858 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:00.859 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:01.122 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.43.5"},{"key":1,"val":"00000000004.58.1"},{"key":2,"val":"00000000006.74.5"},{"key":3,"val":"00000000004.54.1"},{"key":4,"val":"00000000006.50.5"},{"key":5,"val":"00000000004.64.1"},{"key":6,"val":"00000000004.45.1"},{"key":7,"val":"00000000006.61.5"},{"key":8,"val":"00000000004.81.1"},{"key":9,"val":"00000000004.95.1"},{"key":10,"val":"00000000004.71.1"}]} 2020-07-23T12:49:01.123 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-60 2020-07-23T12:49:01.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:01.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:01.125 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:01.171 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:01.171 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:01.546 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.43.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.74.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.50.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.61.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.81.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.95.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}}] 2020-07-23T12:49:01.547 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-60 2020-07-23T12:49:01.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:01.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:01.554 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:01.630 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:01.631 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:01.926 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000002.39.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.93.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.51.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.44.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:01.927 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-60 2020-07-23T12:49:01.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:01.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:01.929 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:01.973 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:01.974 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:02.266 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.39.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.93.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.51.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.44.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:02.266 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-60 2020-07-23T12:49:02.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:02.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:02.268 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:02.314 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:02.315 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:02.574 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000002.39.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.93.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.51.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.44.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:02.575 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-60 2020-07-23T12:49:02.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:02.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:02.577 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:02.624 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:02.624 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:02.905 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.39.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.93.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.51.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.44.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:02.906 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-60 2020-07-23T12:49:02.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:02.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:02.909 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:02.952 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:02.965 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:03.214 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000002.33.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.78.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.47.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.45.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:03.214 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-60 2020-07-23T12:49:03.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:03.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:03.216 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:03.308 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:03.309 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:03.604 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.33.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.78.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.47.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.45.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:03.612 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-60 2020-07-23T12:49:03.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:03.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:03.614 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:03.655 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:03.656 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:03.913 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000002.33.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.78.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.47.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.45.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:03.914 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-60 2020-07-23T12:49:03.915 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:03.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:03.916 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:03.962 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:03.963 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:04.246 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.33.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.78.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.47.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.45.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:04.247 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-60 2020-07-23T12:49:04.247 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-60 zones={test-zone2, test-zone1} 2020-07-23T12:49:05.536 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:49:05.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:05.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:05.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:05.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:05.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:49:05.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:49:05.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:49:05.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:49:05.548 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:49:05.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:05.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:05.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:05.558 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:05.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:49:05.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:49:05.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:49:05.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:49:05.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:05.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:05.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:05.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:05.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:06.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:06.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:06.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:06.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:06.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:06.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:06.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:06.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:06.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:06.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:07.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:07.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:07.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:07.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:07.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:07.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:07.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:07.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:07.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:07.912 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-60 zones={test-zone2, test-zone1} 2020-07-23T12:49:07.913 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-60 zones={test-zone3, test-zone1} 2020-07-23T12:49:08.502 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:49:08.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:08.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:08.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:08.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:08.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:49:08.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:49:08.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:49:08.512 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:49:08.513 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:49:08.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:08.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:08.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:08.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:08.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:49:08.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:49:08.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:49:08.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:49:08.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:08.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:08.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:08.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:08.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:09.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:09.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:09.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:09.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:09.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:09.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:09.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:09.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:09.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:09.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:10.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:10.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:10.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:10.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:10.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:10.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:10.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:10.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:10.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:10.899 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-60 zones={test-zone3, test-zone1} 2020-07-23T12:49:10.901 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-60 zones={test-zone3, test-zone2} 2020-07-23T12:49:11.693 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:49:11.703 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:11.704 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:11.704 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:11.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:11.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:49:11.706 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:49:11.706 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:49:11.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:49:11.707 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:49:11.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:11.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:11.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:11.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:11.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:49:11.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:49:11.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:49:11.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:49:11.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:11.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:11.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:12.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:12.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:12.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:12.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:12.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:12.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:12.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:12.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:12.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:12.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:12.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:13.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:13.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:13.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:13.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:13.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:13.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:13.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:13.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:13.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:13.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:14.038 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-60 zones={test-zone3, test-zone2} 2020-07-23T12:49:14.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:14.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:14.040 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:14.082 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:14.083 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:14.351 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.66.5"},{"key":1,"val":"00000000004.62.1"},{"key":2,"val":"00000000006.64.5"},{"key":3,"val":"00000000004.46.1"},{"key":4,"val":"00000000006.49.5"},{"key":5,"val":"00000000004.75.1"},{"key":6,"val":"00000000004.62.1"},{"key":7,"val":"00000000006.87.5"},{"key":8,"val":"00000000004.62.1"},{"key":9,"val":"00000000004.70.1"},{"key":10,"val":"00000000004.23.1"}]} 2020-07-23T12:49:14.351 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-61 2020-07-23T12:49:14.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:14.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:14.353 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:14.397 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:14.398 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:14.674 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.66.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.64.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.46.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.49.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.87.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.23.1"}}}] 2020-07-23T12:49:14.675 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-61 2020-07-23T12:49:14.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:14.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:14.677 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:14.719 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:14.719 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:14.993 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.66.5"},{"key":1,"val":"00000000004.62.1"},{"key":2,"val":"00000000006.64.5"},{"key":3,"val":"00000000004.46.1"},{"key":4,"val":"00000000006.49.5"},{"key":5,"val":"00000000004.75.1"},{"key":6,"val":"00000000004.62.1"},{"key":7,"val":"00000000006.87.5"},{"key":8,"val":"00000000004.62.1"},{"key":9,"val":"00000000004.70.1"},{"key":10,"val":"00000000004.23.1"}]} 2020-07-23T12:49:14.993 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-61 2020-07-23T12:49:14.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:14.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:14.996 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:15.078 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:15.078 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:15.358 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.66.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.64.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.46.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.49.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.87.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.23.1"}}}] 2020-07-23T12:49:15.359 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-61 2020-07-23T12:49:15.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:15.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:15.362 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:15.444 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:15.444 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:15.718 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.66.5"},{"key":1,"val":"00000000004.62.1"},{"key":2,"val":"00000000006.64.5"},{"key":3,"val":"00000000004.46.1"},{"key":4,"val":"00000000006.49.5"},{"key":5,"val":"00000000004.75.1"},{"key":6,"val":"00000000004.62.1"},{"key":7,"val":"00000000006.87.5"},{"key":8,"val":"00000000004.62.1"},{"key":9,"val":"00000000004.70.1"},{"key":10,"val":"00000000004.23.1"}]} 2020-07-23T12:49:15.719 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-61 2020-07-23T12:49:15.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:15.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:15.722 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:15.769 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:15.769 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:16.048 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.66.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.64.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.46.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.49.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.87.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.23.1"}}}] 2020-07-23T12:49:16.048 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-61 2020-07-23T12:49:16.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:16.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:16.050 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:16.128 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:16.128 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:16.393 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000002.64.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.58.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.50.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.75.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:16.394 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-61 2020-07-23T12:49:16.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:16.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:16.396 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:16.442 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:16.443 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:16.734 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.64.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.58.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.50.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.75.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:16.735 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-61 2020-07-23T12:49:16.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:16.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:16.737 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:16.778 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:16.778 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:17.036 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000002.64.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.58.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.50.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.75.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:17.038 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-61 2020-07-23T12:49:17.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:17.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:17.043 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:17.092 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:17.092 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:17.370 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.64.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.58.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.50.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.75.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:17.371 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-61 2020-07-23T12:49:17.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:17.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:17.373 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:17.455 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:17.456 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:17.728 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000002.66.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.57.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.53.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.68.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:17.728 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-61 2020-07-23T12:49:17.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:17.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:17.730 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:17.813 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:17.813 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:18.087 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.66.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.57.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.53.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.68.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:18.088 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-61 2020-07-23T12:49:18.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:18.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:18.090 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:18.175 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:18.175 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:18.428 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000002.66.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.57.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.53.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.68.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:18.428 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-61 2020-07-23T12:49:18.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:18.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:18.430 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:18.490 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:18.490 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:18.759 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.66.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.57.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.53.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.68.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:18.760 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-61 2020-07-23T12:49:18.760 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-61 zones={test-zone2, test-zone1} 2020-07-23T12:49:20.620 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:49:20.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:20.638 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:20.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:20.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:20.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:49:20.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:49:20.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:49:20.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:49:20.644 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:49:20.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:20.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:20.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:20.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:20.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:49:20.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:49:20.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:49:20.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:49:20.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:20.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:20.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:20.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:21.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:21.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:21.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:21.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:21.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:21.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:21.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:21.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:21.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:21.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:22.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:22.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:22.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:22.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:22.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:22.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:22.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:22.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:22.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:22.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:23.037 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-61 zones={test-zone2, test-zone1} 2020-07-23T12:49:23.038 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-61 zones={test-zone3, test-zone1} 2020-07-23T12:49:23.855 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:49:23.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:23.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:23.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:23.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:23.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:49:23.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:49:23.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:49:23.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:49:23.876 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:49:23.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:23.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:23.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:23.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:23.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:49:23.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:49:23.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:49:23.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:49:23.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:24.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:24.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:24.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:24.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:24.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:24.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:24.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:24.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:25.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:25.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:25.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:25.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:25.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:25.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:25.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:25.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:25.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:25.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:25.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:26.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:26.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:26.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:26.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:26.463 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-61 zones={test-zone3, test-zone1} 2020-07-23T12:49:26.463 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-61 zones={test-zone3, test-zone2} 2020-07-23T12:49:27.313 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:49:27.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:27.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:27.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:27.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:27.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:49:27.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:49:27.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:49:27.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:49:27.329 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:49:27.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:27.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:27.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:27.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:27.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:49:27.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:49:27.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:49:27.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:49:27.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:27.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:27.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:27.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:27.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:27.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:27.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:27.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:28.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:28.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:28.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:28.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:28.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:28.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:28.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:49:28.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:28.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:28.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:49:29.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:29.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:29.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:49:29.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:29.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:29.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:49:29.636 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-61 zones={test-zone3, test-zone2} 2020-07-23T12:49:29.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:29.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:29.638 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:29.677 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:29.677 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:29.964 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595508509.559988_423.1","last_update":"2020-07-23T12:48:29.559988Z"},{"marker":"1_1595508509.891393_677.1","last_update":"2020-07-23T12:48:29.891393Z"},{"marker":"1_1595508509.643677_517.1","last_update":"2020-07-23T12:48:29.643677Z"},{"marker":"1_1595508507.143006_414.3","last_update":"2020-07-23T12:48:27.143006Z"}] 2020-07-23T12:49:29.965 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-23T12:49:29.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:29.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:29.967 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:30.013 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:30.013 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:30.272 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":988729214735988972},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508509.559988_423.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:29.559988Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508509.891393_677.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:29.891393Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508509.643677_517.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:29.643677Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508507.143006_414.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:27.143006Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:49:30.273 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-23T12:49:30.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:30.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:30.276 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:30.315 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:30.315 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:30.617 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595508509.559988_423.1","last_update":"2020-07-23T12:48:29.559988Z"},{"marker":"1_1595508509.891393_677.1","last_update":"2020-07-23T12:48:29.891393Z"},{"marker":"1_1595508509.643677_517.1","last_update":"2020-07-23T12:48:29.643677Z"},{"marker":"1_1595508507.143006_414.3","last_update":"2020-07-23T12:48:27.143006Z"}] 2020-07-23T12:49:30.618 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-23T12:49:30.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:30.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:30.622 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:30.703 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:30.703 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:30.970 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":5372564157323408779},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508509.559988_423.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:29.559988Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508509.891393_677.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:29.891393Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508509.643677_517.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:29.643677Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508507.143006_414.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:27.143006Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:49:30.971 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-23T12:49:30.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:30.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:30.973 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:31.054 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:31.054 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:31.330 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595508509.559988_423.1","last_update":"2020-07-23T12:48:29.559988Z"},{"marker":"1_1595508509.891393_677.1","last_update":"2020-07-23T12:48:29.891393Z"},{"marker":"1_1595508509.643677_517.1","last_update":"2020-07-23T12:48:29.643677Z"},{"marker":"1_1595508507.143006_414.3","last_update":"2020-07-23T12:48:27.143006Z"}] 2020-07-23T12:49:31.330 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-23T12:49:31.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:31.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:31.332 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:31.373 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:31.373 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:31.651 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1496412237659215512},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508509.559988_423.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:29.559988Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508509.891393_677.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:29.891393Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508509.643677_517.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:29.643677Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508507.143006_414.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:27.143006Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:49:31.652 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-23T12:49:31.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:31.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:31.659 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:31.730 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:31.730 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:32.000 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1595508510.229926_312.3","last_update":"2020-07-23T12:48:30.229926Z"},{"marker":"1_1595508510.229926_650.2","last_update":"2020-07-23T12:48:30.229926Z"},{"marker":"1_1595508510.229926_381.3","last_update":"2020-07-23T12:48:30.229926Z"},{"marker":"1_1595508510.229926_384.3","last_update":"2020-07-23T12:48:30.229926Z"}] 2020-07-23T12:49:32.001 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-23T12:49:32.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:32.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:32.003 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:32.052 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:32.053 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:32.317 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14254013310941169947},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508510.229926_312.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:30.229926Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508510.229926_650.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:30.229926Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508510.229926_381.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:30.229926Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508510.229926_384.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:30.229926Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:49:32.317 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-23T12:49:32.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:32.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:32.319 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:32.360 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:32.361 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:32.626 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1595508510.229926_312.3","last_update":"2020-07-23T12:48:30.229926Z"},{"marker":"1_1595508510.229926_650.2","last_update":"2020-07-23T12:48:30.229926Z"},{"marker":"1_1595508510.229926_381.3","last_update":"2020-07-23T12:48:30.229926Z"},{"marker":"1_1595508510.229926_384.3","last_update":"2020-07-23T12:48:30.229926Z"}] 2020-07-23T12:49:32.626 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-23T12:49:32.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:32.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:32.628 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:32.670 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:32.670 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:32.946 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11610391562858473267},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508510.229926_312.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:30.229926Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508510.229926_650.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:30.229926Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508510.229926_381.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:30.229926Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508510.229926_384.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:30.229926Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:49:32.947 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-23T12:49:32.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:32.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:32.948 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:33.026 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:33.027 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:33.312 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595508509.512630_298.1","last_update":"2020-07-23T12:48:29.512630Z"},{"marker":"1_1595508509.458761_547.1","last_update":"2020-07-23T12:48:29.458761Z"},{"marker":"1_1595508509.405835_353.1","last_update":"2020-07-23T12:48:29.405835Z"},{"marker":"1_1595508434.300562_269.1","last_update":"2020-07-23T12:47:14.300562Z"}] 2020-07-23T12:49:33.312 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-23T12:49:33.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:33.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:33.314 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:33.394 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:33.396 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:33.676 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14010726936006743351},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508509.512630_298.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:29.512630Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508509.458761_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:29.458761Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508509.405835_353.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:29.405835Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508434.300562_269.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:47:14.300562Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:49:33.677 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-23T12:49:33.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:33.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:33.679 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:33.754 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:33.754 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:34.032 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595508509.512630_298.1","last_update":"2020-07-23T12:48:29.512630Z"},{"marker":"1_1595508509.458761_547.1","last_update":"2020-07-23T12:48:29.458761Z"},{"marker":"1_1595508509.405835_353.1","last_update":"2020-07-23T12:48:29.405835Z"},{"marker":"1_1595508434.300562_269.1","last_update":"2020-07-23T12:47:14.300562Z"}] 2020-07-23T12:49:34.033 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-23T12:49:34.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:34.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:34.035 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:34.082 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:34.082 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:34.347 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3438718294491381619},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508509.512630_298.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:29.512630Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508509.458761_547.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:29.458761Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508509.405835_353.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:48:29.405835Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508434.300562_269.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:47:14.300562Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:49:34.348 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-23T12:49:34.349 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2020-07-23T12:49:34.350 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-62 2020-07-23T12:49:35.109 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-63 2020-07-23T12:49:36.074 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-64 2020-07-23T12:49:38.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:38.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:38.825 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:38.861 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:38.862 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:39.117 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=a2b03b35-661e-4182-91b6-d60b3b65f694 2020-07-23T12:49:39.171 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595508576.434656_280.1","last_update":"2020-07-23T12:49:36.434656Z"}] 2020-07-23T12:49:39.171 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:49:39.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:39.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:39.173 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:39.222 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:39.223 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:39.477 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508576.434656_280.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:49:36.434656Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:49:39.478 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:49:39.478 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:49:39.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:39.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:39.480 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:39.530 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:39.530 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:39.784 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508576.434656_280.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:49:36.434656Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:49:39.784 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:49:39.785 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:49:39.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:39.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:39.789 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:39.829 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:39.830 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:40.112 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508576.434656_280.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:49:36.434656Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:49:40.113 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:49:40.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:40.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:40.114 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:40.186 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:40.186 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:40.435 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.54.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.44.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.82.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.64.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:40.435 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-62 2020-07-23T12:49:40.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:40.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:40.437 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:40.487 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:40.487 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:40.731 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.82.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:40.731 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-62 2020-07-23T12:49:40.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:40.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:40.734 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:40.777 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:40.777 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:41.034 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.54.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.44.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.82.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.64.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:41.035 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-62 2020-07-23T12:49:41.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:41.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:41.037 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:41.130 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:41.130 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:41.397 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.82.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:41.397 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-62 2020-07-23T12:49:41.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:41.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:41.400 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:41.484 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:41.485 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:41.736 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.54.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.44.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.82.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.64.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:41.737 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-62 2020-07-23T12:49:41.738 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:41.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:41.739 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:41.784 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:41.785 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:42.162 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.54.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.44.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.82.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.64.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:42.163 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-62 2020-07-23T12:49:42.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:42.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:42.166 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:42.255 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:42.256 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:42.517 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000001.55.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.37.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.92.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.44.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:42.518 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-62 2020-07-23T12:49:42.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:42.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:42.521 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:42.564 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:42.565 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:42.853 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.92.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:42.854 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-62 2020-07-23T12:49:42.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:42.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:42.856 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:42.906 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:42.906 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:43.168 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000001.55.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.37.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.92.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.44.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:43.169 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-62 2020-07-23T12:49:43.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:43.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:43.172 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:43.222 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:43.222 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:43.510 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.92.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:43.510 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-62 2020-07-23T12:49:43.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:43.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:43.513 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:43.592 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:43.592 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:43.866 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.59.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.44.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.76.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.52.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:43.867 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-62 2020-07-23T12:49:43.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:43.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:43.868 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:43.951 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:43.952 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:44.253 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:44.254 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-62 2020-07-23T12:49:44.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:44.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:44.256 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:44.339 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:44.339 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:44.598 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.59.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.44.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.76.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.52.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:44.599 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-62 2020-07-23T12:49:44.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:44.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:44.601 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:44.650 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:44.651 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:44.918 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:44.919 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-62 2020-07-23T12:49:44.920 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-62 zones={test-zone2, test-zone1} 2020-07-23T12:49:46.351 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:49:46.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:46.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:46.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:46.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:46.371 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:49:46.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:46.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:46.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:46.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:46.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:46.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:46.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:46.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:46.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:46.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:46.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:47.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:47.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:47.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:47.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:47.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:47.599 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-62 zones={test-zone2, test-zone1} 2020-07-23T12:49:47.600 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-62 zones={test-zone3, test-zone1} 2020-07-23T12:49:48.573 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:49:48.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:48.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:48.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:48.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:48.593 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:49:48.609 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:48.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:48.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:48.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:48.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:48.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:48.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:48.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:49.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:49.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:49.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:49.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:49.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:49.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:49.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:49.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:49.873 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-62 zones={test-zone3, test-zone1} 2020-07-23T12:49:49.874 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-62 zones={test-zone3, test-zone2} 2020-07-23T12:49:50.170 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:49:50.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:50.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:50.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:50.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:50.184 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:49:50.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:50.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:50.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:50.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:50.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:50.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:50.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:50.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:50.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:50.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:50.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:50.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:50.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:51.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:51.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:51.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:51.886 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-62 zones={test-zone3, test-zone2} 2020-07-23T12:49:51.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:51.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:51.888 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:51.935 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:51.935 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:52.194 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.100.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.102.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.84.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.67.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:52.194 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-63 2020-07-23T12:49:52.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:52.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:52.196 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:52.250 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:52.255 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:52.520 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.100.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.102.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.84.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:52.521 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-63 2020-07-23T12:49:52.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:52.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:52.522 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:52.569 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:52.570 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:52.838 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.100.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.102.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.84.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.67.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:52.839 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-63 2020-07-23T12:49:52.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:52.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:52.841 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:52.927 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:52.927 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:53.200 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.100.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.102.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.84.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:53.201 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-63 2020-07-23T12:49:53.202 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:53.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:53.204 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:53.283 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:53.283 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:53.553 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.100.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.102.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.84.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.67.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:53.554 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-63 2020-07-23T12:49:53.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:53.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:53.556 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:53.598 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:53.599 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:53.963 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.100.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.102.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.84.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.67.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:53.964 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-63 2020-07-23T12:49:53.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:53.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:53.967 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:54.044 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:54.044 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:54.300 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000001.110.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.108.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.94.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.85.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:54.301 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-63 2020-07-23T12:49:54.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:54.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:54.302 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:54.347 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:54.348 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:54.654 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.110.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.108.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.94.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:54.655 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-63 2020-07-23T12:49:54.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:54.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:54.658 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:54.714 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:54.715 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:54.984 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000001.110.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.108.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.94.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.85.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:54.985 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-63 2020-07-23T12:49:54.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:54.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:54.987 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:55.027 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:55.028 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:55.303 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.110.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.108.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.94.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:55.304 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-63 2020-07-23T12:49:55.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:55.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:55.307 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:55.386 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:55.387 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:55.653 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.110.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.112.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.78.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.101.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:55.653 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-63 2020-07-23T12:49:55.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:55.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:55.655 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:55.741 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:55.741 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:56.036 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.110.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.112.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.101.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:56.037 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-63 2020-07-23T12:49:56.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:56.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:56.039 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:56.116 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:56.117 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:56.397 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.110.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.112.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.78.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.101.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:49:56.398 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-63 2020-07-23T12:49:56.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:56.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:49:56.406 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:49:56.451 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:49:56.451 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:49:56.725 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.110.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.112.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.101.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:49:56.726 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-63 2020-07-23T12:49:56.727 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-63 zones={test-zone2, test-zone1} 2020-07-23T12:49:57.411 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:49:57.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:57.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:57.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:57.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:57.421 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:49:57.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:57.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:57.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:57.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:57.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:57.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:57.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:57.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:57.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:57.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:58.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:58.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:58.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:58.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:58.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:58.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:58.605 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-63 zones={test-zone2, test-zone1} 2020-07-23T12:49:58.606 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-63 zones={test-zone3, test-zone1} 2020-07-23T12:49:58.754 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:49:58.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:58.762 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:58.762 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:58.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:58.763 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:49:58.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:49:58.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:49:58.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:49:58.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:49:58.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:58.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:58.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:49:59.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:59.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:59.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:49:59.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:59.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:59.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:49:59.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:59.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:59.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:49:59.947 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-63 zones={test-zone3, test-zone1} 2020-07-23T12:49:59.948 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-63 zones={test-zone3, test-zone2} 2020-07-23T12:50:00.082 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:50:00.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:50:00.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:50:00.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:50:00.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:50:00.096 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:50:00.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:50:00.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:50:00.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:50:00.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:50:00.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:50:00.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:50:00.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:50:00.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:50:00.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:50:00.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:50:00.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:50:00.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:50:00.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:50:01.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:50:01.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:50:01.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:50:01.298 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-63 zones={test-zone3, test-zone2} 2020-07-23T12:50:01.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:01.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:01.300 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:01.342 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:01.343 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:01.598 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.43.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.54.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.48.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.51.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:50:01.599 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-64 2020-07-23T12:50:01.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:01.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:01.602 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:01.650 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:01.650 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:01.938 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:50:01.939 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-64 2020-07-23T12:50:01.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:01.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:01.945 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:01.986 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:01.986 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:02.240 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.43.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.54.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.48.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.51.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:50:02.240 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-64 2020-07-23T12:50:02.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:02.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:02.242 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:02.325 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:02.325 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:02.605 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:50:02.606 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-64 2020-07-23T12:50:02.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:02.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:02.609 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:02.688 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:02.688 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:02.976 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.43.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.54.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.48.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.51.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:50:02.977 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-64 2020-07-23T12:50:02.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:02.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:02.979 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:03.022 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:03.022 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:03.298 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.43.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.54.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.48.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.51.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:50:03.299 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-64 2020-07-23T12:50:03.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:03.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:03.301 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:03.390 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:03.391 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:03.650 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000001.43.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.55.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.43.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.53.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:50:03.651 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-64 2020-07-23T12:50:03.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:03.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:03.654 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:03.698 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:03.699 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:03.976 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:50:03.977 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-64 2020-07-23T12:50:03.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:03.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:03.979 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:04.020 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:04.020 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:04.274 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000001.43.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.55.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.43.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.53.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:50:04.275 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-64 2020-07-23T12:50:04.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:04.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:04.277 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:04.324 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:04.325 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:04.601 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:50:04.602 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-64 2020-07-23T12:50:04.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:04.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:04.604 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:04.654 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:04.654 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:04.936 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.39.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.53.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.46.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.49.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:50:04.937 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-64 2020-07-23T12:50:04.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:04.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:04.940 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:05.021 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:05.021 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:05.338 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:50:05.339 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-64 2020-07-23T12:50:05.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:05.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:05.341 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:05.423 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:05.423 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:05.706 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000001.39.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.53.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.46.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.49.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:50:05.707 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-64 2020-07-23T12:50:05.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:05.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:05.708 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:05.751 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:05.752 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:06.061 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:50:06.062 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-64 2020-07-23T12:50:06.063 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-64 zones={test-zone2, test-zone1} 2020-07-23T12:50:07.337 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:50:07.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:50:07.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:50:07.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:50:07.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:50:07.346 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:50:07.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:50:07.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:50:07.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:50:07.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:50:07.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:50:07.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:50:07.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:50:07.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:50:07.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:50:07.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:50:07.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:50:08.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:50:08.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:50:08.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:50:08.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:50:08.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:50:08.543 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-64 zones={test-zone2, test-zone1} 2020-07-23T12:50:08.543 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-64 zones={test-zone3, test-zone1} 2020-07-23T12:50:09.344 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:50:09.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:50:09.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:50:09.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:50:09.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:50:09.355 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:50:09.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:50:09.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:50:09.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:50:09.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:50:09.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:50:09.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:50:09.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:50:09.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:50:09.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:50:09.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:50:09.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:50:10.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:50:10.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:50:10.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:50:10.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:50:10.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:50:10.577 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-64 zones={test-zone3, test-zone1} 2020-07-23T12:50:10.578 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-64 zones={test-zone3, test-zone2} 2020-07-23T12:50:11.284 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:50:11.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:50:11.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:50:11.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:50:11.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:50:11.297 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:50:11.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:50:11.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:50:11.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:50:11.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:50:11.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:50:11.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:50:11.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:50:11.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:50:11.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:50:11.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:50:11.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:50:11.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:50:12.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:50:12.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:50:12.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:50:12.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:50:12.442 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-64 zones={test-zone3, test-zone2} 2020-07-23T12:50:12.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'pdivcf-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:50:12.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'pdivcf-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:50:12.444 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket pdivcf-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:50:12.485 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:12.486 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:12.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'pdivcf-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:50:12.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'pdivcf-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:50:12.860 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket pdivcf-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:50:12.944 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:12.944 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:13.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'pdivcf-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:50:13.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'pdivcf-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:50:13.305 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket pdivcf-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:50:13.386 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:13.387 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:13.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:13.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:13.781 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:13.858 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:13.858 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:14.074 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=a2b03b35-661e-4182-91b6-d60b3b65f694 2020-07-23T12:50:14.136 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595508613.722932_286.1","last_update":"2020-07-23T12:50:13.722932Z"}] 2020-07-23T12:50:14.147 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:50:14.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:14.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:14.149 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:14.194 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:14.195 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:14.460 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508576.434656_280.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:49:36.434656Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:50:14.461 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595508613.722932_286.1 target=1_1595508576.434656_280.1 2020-07-23T12:50:19.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:19.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:19.465 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:19.519 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:19.519 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:19.773 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508613.722932_286.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:13.722932Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:50:19.774 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:50:19.774 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:50:19.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:19.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:19.776 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:19.827 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:19.828 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:20.096 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508613.722932_286.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:13.722932Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:50:20.097 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:50:20.098 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:50:20.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:20.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:20.100 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:20.146 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:20.146 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:20.421 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508613.722932_286.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:13.722932Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:50:20.422 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:50:22.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'pdivcf-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:50:22.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'pdivcf-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:50:22.840 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket pdivcf-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:50:22.892 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:22.893 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:23.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'pdivcf-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:50:23.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'pdivcf-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:50:23.385 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket pdivcf-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:50:23.470 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:23.470 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:23.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'pdivcf-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:50:23.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'pdivcf-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:50:23.812 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket pdivcf-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:50:23.853 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:23.853 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:24.225 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:24.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:24.227 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:24.309 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:24.310 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:24.579 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.66.1"},{"key":1,"val":"00000000004.30.1"},{"key":2,"val":"00000000006.50.1"},{"key":3,"val":"00000000004.67.1"},{"key":4,"val":"00000000006.94.1"},{"key":5,"val":"00000000004.67.1"},{"key":6,"val":"00000000004.68.1"},{"key":7,"val":"00000000006.72.1"},{"key":8,"val":"00000000004.80.1"},{"key":9,"val":"00000000004.66.1"},{"key":10,"val":"00000000004.72.1"}]} 2020-07-23T12:50:24.580 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-62 2020-07-23T12:50:24.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:24.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:24.582 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:24.638 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:24.639 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:24.903 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.66.1"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.50.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.94.1"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.72.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:50:24.904 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.30.1 target=, shard=5 master=00000000004.67.1 target=, shard=9 master=00000000004.66.1 target= 2020-07-23T12:50:29.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:29.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:29.907 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:29.949 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:29.949 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:30.225 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.66.1"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.50.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.94.1"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.72.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:50:30.226 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.30.1 target=, shard=5 master=00000000004.67.1 target=, shard=9 master=00000000004.66.1 target= 2020-07-23T12:50:35.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:35.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:35.230 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:35.274 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:35.275 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:35.538 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.66.1"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.50.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.94.1"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.72.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:50:35.539 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.30.1 target=, shard=5 master=00000000004.67.1 target=, shard=9 master=00000000004.66.1 target= 2020-07-23T12:50:40.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:40.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:40.543 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:40.583 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:40.584 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:40.848 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.66.1"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.50.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.94.1"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.72.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:50:40.849 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.30.1 target=, shard=5 master=00000000004.67.1 target=, shard=9 master=00000000004.66.1 target= 2020-07-23T12:50:45.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:45.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:45.852 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:45.914 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:45.915 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:46.188 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.66.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.50.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.94.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.72.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:50:46.189 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-62 2020-07-23T12:50:46.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:46.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:46.191 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:46.234 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:46.235 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:46.510 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.66.1"},{"key":1,"val":"00000000004.30.1"},{"key":2,"val":"00000000006.50.1"},{"key":3,"val":"00000000004.67.1"},{"key":4,"val":"00000000006.94.1"},{"key":5,"val":"00000000004.67.1"},{"key":6,"val":"00000000004.68.1"},{"key":7,"val":"00000000006.72.1"},{"key":8,"val":"00000000004.80.1"},{"key":9,"val":"00000000004.66.1"},{"key":10,"val":"00000000004.72.1"}]} 2020-07-23T12:50:46.510 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-62 2020-07-23T12:50:46.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:46.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:46.512 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:46.603 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:46.603 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:46.887 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.66.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.50.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.94.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.72.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:50:46.888 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-62 2020-07-23T12:50:46.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:46.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:46.891 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:46.968 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:46.969 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:47.215 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.66.1"},{"key":1,"val":"00000000004.30.1"},{"key":2,"val":"00000000006.50.1"},{"key":3,"val":"00000000004.67.1"},{"key":4,"val":"00000000006.94.1"},{"key":5,"val":"00000000004.67.1"},{"key":6,"val":"00000000004.68.1"},{"key":7,"val":"00000000006.72.1"},{"key":8,"val":"00000000004.80.1"},{"key":9,"val":"00000000004.66.1"},{"key":10,"val":"00000000004.72.1"}]} 2020-07-23T12:50:47.215 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-62 2020-07-23T12:50:47.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:47.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:47.217 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:47.265 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:47.265 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:47.535 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:50:47.536 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:50:52.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:52.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:52.539 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:52.584 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:52.585 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:52.955 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:50:52.956 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:50:57.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:57.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:50:57.959 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:50:58.014 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:50:58.014 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:50:58.284 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:50:58.284 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:51:03.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:03.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:03.287 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:51:03.332 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:51:03.332 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:51:03.608 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:51:03.609 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:51:08.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:08.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:08.613 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:51:08.658 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:51:08.659 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:51:09.040 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:51:09.041 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:51:14.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:14.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:14.044 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:51:14.090 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:51:14.090 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:51:14.566 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:51:14.567 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:51:19.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:19.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:19.570 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:51:19.614 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:51:19.615 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:51:19.893 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:51:19.894 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:51:24.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:24.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:24.898 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:51:24.949 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:51:24.950 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:51:25.219 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:51:25.220 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:51:30.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:30.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:30.223 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:51:30.266 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:51:30.267 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:51:30.634 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:51:30.635 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:51:35.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:35.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:35.638 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:51:35.683 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:51:35.684 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:51:35.955 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:51:35.956 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:51:40.957 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:40.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:40.959 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:51:41.008 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:51:41.008 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:51:41.279 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:51:41.280 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:51:46.281 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:46.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:46.283 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:51:46.327 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:51:46.328 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:51:46.696 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:51:46.697 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:51:51.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:51.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:51.701 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:51:51.742 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:51:51.743 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:51:52.115 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:51:52.116 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:51:57.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:57.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:51:57.121 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:51:57.167 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:51:57.168 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:51:57.439 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:51:57.440 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:52:02.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:02.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:02.444 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:52:02.489 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:52:02.490 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:52:02.764 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:52:02.765 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:52:07.767 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:07.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:07.769 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:52:07.813 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:52:07.814 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:52:08.187 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:52:08.188 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:52:13.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:13.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:13.191 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:52:13.236 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:52:13.236 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:52:13.509 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:52:13.510 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:52:18.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:18.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:18.513 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:52:18.565 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:52:18.565 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:52:18.816 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:52:18.817 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:52:23.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:23.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:23.820 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:52:23.870 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:52:23.871 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:52:24.237 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:52:24.238 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:52:29.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:29.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:29.241 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:52:29.286 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:52:29.286 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:52:29.559 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:52:29.560 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:52:34.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:34.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:34.564 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:52:34.608 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:52:34.608 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:52:34.976 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:52:34.977 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:52:39.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:39.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:39.981 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:52:40.027 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:52:40.028 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:52:40.405 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:52:40.406 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:52:45.408 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:45.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:45.410 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:52:45.462 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:52:45.463 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:52:45.718 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:52:45.719 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:52:50.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:50.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:50.723 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:52:50.765 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:52:50.766 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:52:51.051 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:52:51.052 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1, shard=7 master=00000000006.72.1 target=00000000002.65.1 2020-07-23T12:52:56.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:56.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:52:56.056 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:52:56.099 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:52:56.099 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:52:56.474 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.72.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:52:56.475 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1 2020-07-23T12:53:01.478 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:01.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:01.483 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:01.528 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:01.529 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:01.780 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.72.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:53:01.781 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1 2020-07-23T12:53:06.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:06.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:06.785 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:06.831 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:06.832 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:07.108 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.85.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.72.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:53:07.108 WARNING:rgw_multi.tests:bucket pdivcf-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.66.1 target=00000000002.55.1, shard=2 master=00000000006.50.1 target=00000000002.45.1, shard=4 master=00000000006.94.1 target=00000000002.85.1 2020-07-23T12:53:12.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:12.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:12.111 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:12.156 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:12.156 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:12.437 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.66.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.50.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.94.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.72.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}}] 2020-07-23T12:53:12.438 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-62 2020-07-23T12:53:12.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:12.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:12.440 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:12.522 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:12.523 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:12.776 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000004.64.1"},{"key":1,"val":"00000000002.29.1"},{"key":2,"val":"00000000004.42.1"},{"key":3,"val":"00000000002.65.1"},{"key":4,"val":"00000000004.107.1"},{"key":5,"val":"00000000002.95.1"},{"key":6,"val":"00000000002.66.1"},{"key":7,"val":"00000000004.52.1"},{"key":8,"val":"00000000002.55.1"},{"key":9,"val":"00000000002.47.1"},{"key":10,"val":"00000000002.66.1"}]} 2020-07-23T12:53:12.776 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-62 2020-07-23T12:53:12.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:12.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:12.778 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:12.826 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:12.830 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:13.108 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.64.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.29.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.42.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.107.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.95.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.66.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.52.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.66.1"}}}] 2020-07-23T12:53:13.109 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-62 2020-07-23T12:53:13.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:13.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:13.116 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:13.158 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:13.159 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:13.423 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000004.64.1"},{"key":1,"val":"00000000002.29.1"},{"key":2,"val":"00000000004.42.1"},{"key":3,"val":"00000000002.65.1"},{"key":4,"val":"00000000004.107.1"},{"key":5,"val":"00000000002.95.1"},{"key":6,"val":"00000000002.66.1"},{"key":7,"val":"00000000004.52.1"},{"key":8,"val":"00000000002.55.1"},{"key":9,"val":"00000000002.47.1"},{"key":10,"val":"00000000002.66.1"}]} 2020-07-23T12:53:13.424 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-62 2020-07-23T12:53:13.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:13.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:13.426 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:13.471 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:13.471 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:13.777 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.64.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.29.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.42.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.107.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.95.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.66.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.52.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.66.1"}}}] 2020-07-23T12:53:13.778 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-62 2020-07-23T12:53:13.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:13.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:13.782 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:13.861 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:13.864 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:14.114 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000004.68.1"},{"key":1,"val":"00000000002.30.1"},{"key":2,"val":"00000000004.49.1"},{"key":3,"val":"00000000002.69.1"},{"key":4,"val":"00000000004.91.1"},{"key":5,"val":"00000000002.83.1"},{"key":6,"val":"00000000002.70.1"},{"key":7,"val":"00000000004.60.1"},{"key":8,"val":"00000000002.48.1"},{"key":9,"val":"00000000002.48.1"},{"key":10,"val":"00000000002.64.1"}]} 2020-07-23T12:53:14.115 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-62 2020-07-23T12:53:14.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:14.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:14.117 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:14.198 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:14.208 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:14.493 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.68.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.30.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.91.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.83.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.70.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.60.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.48.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.48.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.64.1"}}}] 2020-07-23T12:53:14.494 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-62 2020-07-23T12:53:14.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:14.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:14.496 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:14.575 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:14.576 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:14.850 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000004.68.1"},{"key":1,"val":"00000000002.30.1"},{"key":2,"val":"00000000004.49.1"},{"key":3,"val":"00000000002.69.1"},{"key":4,"val":"00000000004.91.1"},{"key":5,"val":"00000000002.83.1"},{"key":6,"val":"00000000002.70.1"},{"key":7,"val":"00000000004.60.1"},{"key":8,"val":"00000000002.48.1"},{"key":9,"val":"00000000002.48.1"},{"key":10,"val":"00000000002.64.1"}]} 2020-07-23T12:53:14.851 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-62 2020-07-23T12:53:14.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:14.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:14.853 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:14.903 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:14.903 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:15.196 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.68.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.30.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.91.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.83.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.70.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.60.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.48.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.48.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.64.1"}}}] 2020-07-23T12:53:15.197 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-62 2020-07-23T12:53:15.197 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-62 zones={test-zone2, test-zone1} 2020-07-23T12:53:15.216 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:53:15.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:15.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:15.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:15.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:15.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:15.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:15.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:15.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:15.228 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:53:15.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:15.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:15.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:15.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:15.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:15.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:15.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:15.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:15.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:15.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:15.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:15.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:15.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:15.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:15.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:15.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:16.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:16.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:16.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:16.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:16.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:16.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:16.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:16.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:16.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:16.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:17.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:17.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:17.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:17.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:17.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:17.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:17.591 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-62 zones={test-zone2, test-zone1} 2020-07-23T12:53:17.592 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-62 zones={test-zone3, test-zone1} 2020-07-23T12:53:17.602 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:53:17.609 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:17.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:17.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:17.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:17.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:17.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:17.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:17.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:17.614 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:53:17.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:17.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:17.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:17.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:17.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:17.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:17.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:17.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:17.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:17.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:17.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:17.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:18.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:18.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:18.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:18.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:18.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:18.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:18.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:18.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:18.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:18.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:19.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:19.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:19.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:19.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:19.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:19.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:19.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:19.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:19.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:19.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:20.032 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-62 zones={test-zone3, test-zone1} 2020-07-23T12:53:20.032 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-62 zones={test-zone3, test-zone2} 2020-07-23T12:53:20.624 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:53:20.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:20.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:20.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:20.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:20.638 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:20.638 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:20.638 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:20.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:20.639 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:53:20.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:20.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:20.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:20.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:20.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:20.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:20.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:20.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:20.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:20.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:20.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:20.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:21.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:21.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:21.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:21.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:21.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:21.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:21.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:21.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:21.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:21.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:21.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:22.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:22.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:22.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:22.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:22.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:22.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:22.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:22.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:22.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:22.974 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-62 zones={test-zone3, test-zone2} 2020-07-23T12:53:22.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:22.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:22.977 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:23.019 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:23.020 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:23.289 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.116.5"},{"key":1,"val":"00000000004.75.1"},{"key":2,"val":"00000000006.118.5"},{"key":3,"val":"00000000004.114.1"},{"key":4,"val":"00000000006.100.5"},{"key":5,"val":"00000000004.53.1"},{"key":6,"val":"00000000004.76.1"},{"key":7,"val":"00000000006.81.5"},{"key":8,"val":"00000000004.74.1"},{"key":9,"val":"00000000004.92.1"},{"key":10,"val":"00000000004.69.1"}]} 2020-07-23T12:53:23.290 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-63 2020-07-23T12:53:23.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:23.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:23.292 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:23.335 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:23.335 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:23.634 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.116.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.118.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.114.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.100.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.81.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1"}}}] 2020-07-23T12:53:23.635 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-63 2020-07-23T12:53:23.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:23.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:23.636 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:23.677 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:23.959 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.116.5"},{"key":1,"val":"00000000004.75.1"},{"key":2,"val":"00000000006.118.5"},{"key":3,"val":"00000000004.114.1"},{"key":4,"val":"00000000006.100.5"},{"key":5,"val":"00000000004.53.1"},{"key":6,"val":"00000000004.76.1"},{"key":7,"val":"00000000006.81.5"},{"key":8,"val":"00000000004.74.1"},{"key":9,"val":"00000000004.92.1"},{"key":10,"val":"00000000004.69.1"}]} 2020-07-23T12:53:23.959 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-63 2020-07-23T12:53:23.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:23.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:23.961 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:24.046 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:24.047 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:24.352 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.116.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.118.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.114.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.100.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.81.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1"}}}] 2020-07-23T12:53:24.353 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-63 2020-07-23T12:53:24.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:24.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:24.355 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:24.393 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:24.394 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:24.687 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.116.5"},{"key":1,"val":"00000000004.75.1"},{"key":2,"val":"00000000006.118.5"},{"key":3,"val":"00000000004.114.1"},{"key":4,"val":"00000000006.100.5"},{"key":5,"val":"00000000004.53.1"},{"key":6,"val":"00000000004.76.1"},{"key":7,"val":"00000000006.81.5"},{"key":8,"val":"00000000004.74.1"},{"key":9,"val":"00000000004.92.1"},{"key":10,"val":"00000000004.69.1"}]} 2020-07-23T12:53:24.688 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-63 2020-07-23T12:53:24.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:24.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:24.691 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:24.734 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:24.734 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:25.097 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.116.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.118.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.114.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.100.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.81.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1"}}}] 2020-07-23T12:53:25.098 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-63 2020-07-23T12:53:25.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:25.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:25.100 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:25.180 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:25.180 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:25.459 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000004.121.1"},{"key":1,"val":"00000000002.81.1"},{"key":2,"val":"00000000004.122.1"},{"key":3,"val":"00000000002.123.1"},{"key":4,"val":"00000000004.105.1"},{"key":5,"val":"00000000002.56.1"},{"key":6,"val":"00000000002.94.1"},{"key":7,"val":"00000000004.95.1"},{"key":8,"val":"00000000002.51.1"},{"key":9,"val":"00000000002.79.1"},{"key":10,"val":"00000000002.66.1"}]} 2020-07-23T12:53:25.460 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-63 2020-07-23T12:53:25.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:25.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:25.462 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:25.506 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:25.506 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:25.811 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.121.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.81.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.122.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.123.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.105.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.94.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.95.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.79.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.66.1"}}}] 2020-07-23T12:53:25.812 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-63 2020-07-23T12:53:25.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:25.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:25.815 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:25.856 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:25.856 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:26.118 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000004.121.1"},{"key":1,"val":"00000000002.81.1"},{"key":2,"val":"00000000004.122.1"},{"key":3,"val":"00000000002.123.1"},{"key":4,"val":"00000000004.105.1"},{"key":5,"val":"00000000002.56.1"},{"key":6,"val":"00000000002.94.1"},{"key":7,"val":"00000000004.95.1"},{"key":8,"val":"00000000002.51.1"},{"key":9,"val":"00000000002.79.1"},{"key":10,"val":"00000000002.66.1"}]} 2020-07-23T12:53:26.119 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-63 2020-07-23T12:53:26.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:26.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:26.121 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:26.163 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:26.164 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:26.462 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.121.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.81.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.122.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.123.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.105.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.94.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.95.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.79.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.66.1"}}}] 2020-07-23T12:53:26.462 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-63 2020-07-23T12:53:26.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:26.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:26.466 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:26.544 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:26.544 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:26.809 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000004.123.1"},{"key":1,"val":"00000000002.84.1"},{"key":2,"val":"00000000004.124.1"},{"key":3,"val":"00000000002.125.1"},{"key":4,"val":"00000000004.89.1"},{"key":5,"val":"00000000002.50.1"},{"key":6,"val":"00000000002.110.1"},{"key":7,"val":"00000000004.111.1"},{"key":8,"val":"00000000002.59.1"},{"key":9,"val":"00000000002.72.1"},{"key":10,"val":"00000000002.73.1"}]} 2020-07-23T12:53:26.809 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-63 2020-07-23T12:53:26.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:26.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:26.811 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:26.903 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:26.903 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:27.186 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.123.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.84.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.124.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.125.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.89.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.50.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.110.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.111.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.59.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.72.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.73.1"}}}] 2020-07-23T12:53:27.187 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-63 2020-07-23T12:53:27.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:27.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:27.188 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:27.269 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:27.270 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:27.538 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000004.123.1"},{"key":1,"val":"00000000002.84.1"},{"key":2,"val":"00000000004.124.1"},{"key":3,"val":"00000000002.125.1"},{"key":4,"val":"00000000004.89.1"},{"key":5,"val":"00000000002.50.1"},{"key":6,"val":"00000000002.110.1"},{"key":7,"val":"00000000004.111.1"},{"key":8,"val":"00000000002.59.1"},{"key":9,"val":"00000000002.72.1"},{"key":10,"val":"00000000002.73.1"}]} 2020-07-23T12:53:27.539 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-63 2020-07-23T12:53:27.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:27.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:27.540 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:27.586 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:27.587 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:27.871 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.123.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.84.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.124.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.125.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.89.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.50.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.110.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.111.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.59.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.72.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.73.1"}}}] 2020-07-23T12:53:27.872 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-63 2020-07-23T12:53:27.872 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-63 zones={test-zone2, test-zone1} 2020-07-23T12:53:28.952 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:53:28.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:28.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:28.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:28.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:28.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:28.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:28.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:28.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:28.965 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:53:29.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:29.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:29.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:29.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:29.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:29.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:29.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:29.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:29.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:29.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:29.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:29.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:29.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:29.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:30.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:30.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:30.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:30.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:30.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:30.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:30.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:30.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:30.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:30.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:30.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:31.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:31.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:31.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:31.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:31.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:31.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:31.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:31.732 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-63 zones={test-zone2, test-zone1} 2020-07-23T12:53:31.733 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-63 zones={test-zone3, test-zone1} 2020-07-23T12:53:31.951 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:53:31.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:31.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:31.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:31.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:31.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:31.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:31.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:31.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:31.961 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:53:31.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:31.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:31.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:31.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:31.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:31.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:31.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:31.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:31.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:32.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:32.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:32.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:32.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:32.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:32.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:32.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:32.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:32.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:32.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:33.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:33.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:33.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:33.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:33.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:33.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:33.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:33.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:34.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:34.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:34.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:34.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:34.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:34.544 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-63 zones={test-zone3, test-zone1} 2020-07-23T12:53:34.545 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-63 zones={test-zone3, test-zone2} 2020-07-23T12:53:35.399 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:53:35.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:35.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:35.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:35.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:35.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:35.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:35.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:35.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:35.425 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:53:35.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:35.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:35.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:35.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:35.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:35.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:35.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:35.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:35.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:35.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:35.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:35.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:35.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:35.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:36.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:36.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:36.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:36.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:36.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:36.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:36.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:36.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:36.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:36.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:36.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:37.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:37.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:37.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:37.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:37.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:37.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:37.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:37.762 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-63 zones={test-zone3, test-zone2} 2020-07-23T12:53:37.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:37.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:37.764 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:37.807 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:37.808 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:38.070 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.49.5"},{"key":1,"val":"00000000004.74.1"},{"key":2,"val":"00000000006.64.5"},{"key":3,"val":"00000000004.63.1"},{"key":4,"val":"00000000006.54.5"},{"key":5,"val":"00000000004.64.1"},{"key":6,"val":"00000000004.71.1"},{"key":7,"val":"00000000006.57.5"},{"key":8,"val":"00000000004.79.1"},{"key":9,"val":"00000000004.98.1"},{"key":10,"val":"00000000004.62.1"}]} 2020-07-23T12:53:38.071 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-64 2020-07-23T12:53:38.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:38.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:38.073 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:38.117 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:38.117 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:38.393 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.49.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.64.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.63.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.54.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.57.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.98.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}}] 2020-07-23T12:53:38.394 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-64 2020-07-23T12:53:38.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:38.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:38.396 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:38.438 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:38.438 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:38.732 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.49.5"},{"key":1,"val":"00000000004.74.1"},{"key":2,"val":"00000000006.64.5"},{"key":3,"val":"00000000004.63.1"},{"key":4,"val":"00000000006.54.5"},{"key":5,"val":"00000000004.64.1"},{"key":6,"val":"00000000004.71.1"},{"key":7,"val":"00000000006.57.5"},{"key":8,"val":"00000000004.79.1"},{"key":9,"val":"00000000004.98.1"},{"key":10,"val":"00000000004.62.1"}]} 2020-07-23T12:53:38.733 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-64 2020-07-23T12:53:38.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:38.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:38.737 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:38.821 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:38.821 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:39.089 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.49.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.64.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.63.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.54.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.57.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.98.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}}] 2020-07-23T12:53:39.101 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-64 2020-07-23T12:53:39.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:39.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:39.103 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:39.177 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:39.178 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:39.459 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000006.49.5"},{"key":1,"val":"00000000004.74.1"},{"key":2,"val":"00000000006.64.5"},{"key":3,"val":"00000000004.63.1"},{"key":4,"val":"00000000006.54.5"},{"key":5,"val":"00000000004.64.1"},{"key":6,"val":"00000000004.71.1"},{"key":7,"val":"00000000006.57.5"},{"key":8,"val":"00000000004.79.1"},{"key":9,"val":"00000000004.98.1"},{"key":10,"val":"00000000004.62.1"}]} 2020-07-23T12:53:39.460 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-64 2020-07-23T12:53:39.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:39.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:39.464 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:39.517 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:39.517 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:39.790 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.49.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.64.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.63.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.54.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.57.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.98.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}}] 2020-07-23T12:53:39.791 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-64 2020-07-23T12:53:39.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:39.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:39.793 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:39.835 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:39.835 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:40.109 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000004.49.5"},{"key":1,"val":"00000000002.63.1"},{"key":2,"val":"00000000004.65.5"},{"key":3,"val":"00000000002.75.1"},{"key":4,"val":"00000000004.49.5"},{"key":5,"val":"00000000002.78.1"},{"key":6,"val":"00000000002.67.1"},{"key":7,"val":"00000000004.59.5"},{"key":8,"val":"00000000002.96.1"},{"key":9,"val":"00000000002.106.1"},{"key":10,"val":"00000000002.61.1"}]} 2020-07-23T12:53:40.110 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-64 2020-07-23T12:53:40.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:40.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:40.112 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:40.156 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:40.157 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:40.460 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.49.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.65.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.75.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.49.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.78.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.67.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.59.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.96.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.106.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}}] 2020-07-23T12:53:40.461 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-64 2020-07-23T12:53:40.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:40.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:40.463 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:40.504 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:40.505 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:40.759 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":"00000000004.49.5"},{"key":1,"val":"00000000002.63.1"},{"key":2,"val":"00000000004.65.5"},{"key":3,"val":"00000000002.75.1"},{"key":4,"val":"00000000004.49.5"},{"key":5,"val":"00000000002.78.1"},{"key":6,"val":"00000000002.67.1"},{"key":7,"val":"00000000004.59.5"},{"key":8,"val":"00000000002.96.1"},{"key":9,"val":"00000000002.106.1"},{"key":10,"val":"00000000002.61.1"}]} 2020-07-23T12:53:40.760 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-64 2020-07-23T12:53:40.761 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:40.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:40.762 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:40.807 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:40.808 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:41.086 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.49.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.65.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.75.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.49.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.78.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.67.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.59.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.96.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.106.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}}] 2020-07-23T12:53:41.087 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-64 2020-07-23T12:53:41.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:41.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:41.089 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:41.170 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:41.170 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:41.443 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000004.44.1"},{"key":1,"val":"00000000002.61.1"},{"key":2,"val":"00000000004.60.1"},{"key":3,"val":"00000000002.75.1"},{"key":4,"val":"00000000004.51.1"},{"key":5,"val":"00000000002.78.1"},{"key":6,"val":"00000000002.71.1"},{"key":7,"val":"00000000004.54.1"},{"key":8,"val":"00000000002.112.1"},{"key":9,"val":"00000000002.90.1"},{"key":10,"val":"00000000002.61.1"}]} 2020-07-23T12:53:41.444 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-64 2020-07-23T12:53:41.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:41.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:41.446 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:41.539 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:41.539 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:41.830 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.44.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.60.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.75.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.51.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.78.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.112.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.90.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}}] 2020-07-23T12:53:41.831 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-64 2020-07-23T12:53:41.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:41.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:41.833 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:41.873 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:41.874 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:42.136 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":"00000000004.44.1"},{"key":1,"val":"00000000002.61.1"},{"key":2,"val":"00000000004.60.1"},{"key":3,"val":"00000000002.75.1"},{"key":4,"val":"00000000004.51.1"},{"key":5,"val":"00000000002.78.1"},{"key":6,"val":"00000000002.71.1"},{"key":7,"val":"00000000004.54.1"},{"key":8,"val":"00000000002.112.1"},{"key":9,"val":"00000000002.90.1"},{"key":10,"val":"00000000002.61.1"}]} 2020-07-23T12:53:42.137 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-64 2020-07-23T12:53:42.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:42.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:42.138 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:42.183 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:42.184 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:42.474 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.44.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.60.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.75.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.51.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.78.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.112.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.90.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}}] 2020-07-23T12:53:42.474 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-64 2020-07-23T12:53:42.475 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-64 zones={test-zone2, test-zone1} 2020-07-23T12:53:43.580 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:53:43.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:43.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:43.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:43.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:43.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:43.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:43.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:43.594 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:43.595 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:53:43.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:43.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:43.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:43.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:43.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:43.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:43.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:43.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:43.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:43.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:43.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:43.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:43.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:44.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:44.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:44.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:44.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:44.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:44.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:44.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:44.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:44.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:44.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:45.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:45.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:45.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:45.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:45.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:45.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:45.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:45.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:45.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:45.899 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-64 zones={test-zone2, test-zone1} 2020-07-23T12:53:45.899 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-64 zones={test-zone3, test-zone1} 2020-07-23T12:53:46.535 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:53:46.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:46.541 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:46.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:46.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:46.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:46.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:46.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:46.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:46.546 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:53:46.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:46.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:46.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:46.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:46.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:46.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:46.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:46.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:46.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:46.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:46.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:46.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:46.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:47.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:47.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:47.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:47.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:47.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:47.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:47.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:47.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:47.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:47.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:48.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:48.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:48.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:48.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:48.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:48.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:48.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:48.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:48.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:48.845 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-64 zones={test-zone3, test-zone1} 2020-07-23T12:53:48.846 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-64 zones={test-zone3, test-zone2} 2020-07-23T12:53:49.461 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:53:49.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:49.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:49.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:49.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:49.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:49.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:49.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:49.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:49.478 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:53:49.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-07-23T12:53:49.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-07-23T12:53:49.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-07-23T12:53:49.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-07-23T12:53:49.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-07-23T12:53:49.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-07-23T12:53:49.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-07-23T12:53:49.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-07-23T12:53:49.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:49.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:49.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-07-23T12:53:49.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:49.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:49.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-07-23T12:53:50.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:50.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:50.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-07-23T12:53:50.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:50.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:50.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-07-23T12:53:50.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:50.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:50.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-07-23T12:53:50.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:50.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:51.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-07-23T12:53:51.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:51.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:51.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-07-23T12:53:51.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:51.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:51.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-07-23T12:53:51.776 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-64 zones={test-zone3, test-zone2} 2020-07-23T12:53:51.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:51.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:51.778 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:51.819 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:51.819 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:52.093 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595508628.859949_594.1","last_update":"2020-07-23T12:50:28.859949Z"},{"marker":"1_1595508628.811522_749.1","last_update":"2020-07-23T12:50:28.811522Z"},{"marker":"1_1595508628.783092_631.1","last_update":"2020-07-23T12:50:28.783092Z"},{"marker":"1_1595508624.186260_592.3","last_update":"2020-07-23T12:50:24.186260Z"}] 2020-07-23T12:53:52.094 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-23T12:53:52.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:52.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:52.096 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:52.137 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:52.137 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:52.413 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":988729214735988972},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508628.859949_594.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:28.859949Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508628.811522_749.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:28.811522Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508628.783092_631.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:28.783092Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508624.186260_592.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:24.186260Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:53:52.413 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-07-23T12:53:52.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:52.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:52.415 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:52.453 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:52.454 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:52.739 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595508628.859949_594.1","last_update":"2020-07-23T12:50:28.859949Z"},{"marker":"1_1595508628.811522_749.1","last_update":"2020-07-23T12:50:28.811522Z"},{"marker":"1_1595508628.783092_631.1","last_update":"2020-07-23T12:50:28.783092Z"},{"marker":"1_1595508624.186260_592.3","last_update":"2020-07-23T12:50:24.186260Z"}] 2020-07-23T12:53:52.739 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-23T12:53:52.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:52.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:52.741 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:52.822 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:52.822 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:53.080 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":5372564157323408779},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508628.859949_594.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:28.859949Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508628.811522_749.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:28.811522Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508628.783092_631.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:28.783092Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508624.186260_592.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:24.186260Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:53:53.080 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-07-23T12:53:53.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:53.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:53.082 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:53.163 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:53.163 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:53.431 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595508628.859949_594.1","last_update":"2020-07-23T12:50:28.859949Z"},{"marker":"1_1595508628.811522_749.1","last_update":"2020-07-23T12:50:28.811522Z"},{"marker":"1_1595508628.783092_631.1","last_update":"2020-07-23T12:50:28.783092Z"},{"marker":"1_1595508624.186260_592.3","last_update":"2020-07-23T12:50:24.186260Z"}] 2020-07-23T12:53:53.431 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-23T12:53:53.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:53.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:53.433 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:53.475 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:53.476 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:53.751 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1496412237659215512},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508628.859949_594.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:28.859949Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508628.811522_749.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:28.811522Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508628.783092_631.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:28.783092Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508624.186260_592.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:24.186260Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:53:53.752 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-07-23T12:53:53.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:53.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:53.754 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:53.832 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:53.832 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:54.100 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1595508642.249984_453.9","last_update":"2020-07-23T12:50:42.249984Z"},{"marker":"1_1595508642.249984_701.6","last_update":"2020-07-23T12:50:42.249984Z"},{"marker":"1_1595508642.249984_456.9","last_update":"2020-07-23T12:50:42.249984Z"},{"marker":"1_1595508642.249984_634.9","last_update":"2020-07-23T12:50:42.249984Z"}] 2020-07-23T12:53:54.101 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-23T12:53:54.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:54.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:54.102 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:54.149 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:54.150 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:54.428 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14254013310941169947},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508642.249984_453.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:42.249984Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508642.249984_701.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:42.249984Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508642.249984_456.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:42.249984Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508642.249984_634.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:42.249984Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:53:54.429 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-07-23T12:53:54.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:54.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:54.431 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:54.471 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:54.471 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:54.724 INFO:teuthology.orchestra.run.smithi133.stdout:[{"marker":"1_1595508642.249984_453.9","last_update":"2020-07-23T12:50:42.249984Z"},{"marker":"1_1595508642.249984_701.6","last_update":"2020-07-23T12:50:42.249984Z"},{"marker":"1_1595508642.249984_456.9","last_update":"2020-07-23T12:50:42.249984Z"},{"marker":"1_1595508642.249984_634.9","last_update":"2020-07-23T12:50:42.249984Z"}] 2020-07-23T12:53:54.724 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-23T12:53:54.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:54.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:54.726 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:54.769 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:54.770 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:55.040 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11610391562858473267},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508642.249984_453.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:42.249984Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508642.249984_701.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:42.249984Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508642.249984_456.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:42.249984Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508642.249984_634.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:42.249984Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:53:55.041 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-07-23T12:53:55.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:55.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:55.052 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:55.120 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:55.120 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:55.399 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595508640.686347_456.9","last_update":"2020-07-23T12:50:40.686347Z"},{"marker":"1_1595508640.686347_597.6","last_update":"2020-07-23T12:50:40.686347Z"},{"marker":"1_1595508640.686347_438.9","last_update":"2020-07-23T12:50:40.686347Z"},{"marker":"1_1595508640.686347_480.9","last_update":"2020-07-23T12:50:40.686347Z"}] 2020-07-23T12:53:55.399 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-23T12:53:55.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:55.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:55.401 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:55.481 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:55.481 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:55.761 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14010726936006743351},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508640.686347_456.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:40.686347Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508640.686347_597.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:40.686347Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508640.686347_438.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:40.686347Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508640.686347_480.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:40.686347Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:53:55.763 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-07-23T12:53:55.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:55.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:55.764 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:55.844 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:55.845 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:56.107 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"1_1595508640.686347_456.9","last_update":"2020-07-23T12:50:40.686347Z"},{"marker":"1_1595508640.686347_597.6","last_update":"2020-07-23T12:50:40.686347Z"},{"marker":"1_1595508640.686347_438.9","last_update":"2020-07-23T12:50:40.686347Z"},{"marker":"1_1595508640.686347_480.9","last_update":"2020-07-23T12:50:40.686347Z"}] 2020-07-23T12:53:56.108 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-23T12:53:56.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:56.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:53:56.110 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:53:56.153 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:53:56.153 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:53:56.416 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3438718294491381619},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1595508640.686347_456.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:40.686347Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1595508640.686347_597.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:40.686347Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1595508640.686347_438.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:40.686347Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1595508640.686347_480.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:50:40.686347Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-07-23T12:53:56.417 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-07-23T12:53:56.419 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2020-07-23T12:53:56.420 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-65 2020-07-23T12:53:56.762 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=pdivcf-66 2020-07-23T12:53:57.613 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=pdivcf-67 2020-07-23T12:54:01.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:01.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:01.012 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:01.051 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:01.052 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:01.346 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.64.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:54:01.347 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-65 2020-07-23T12:54:01.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:01.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:01.349 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:01.394 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:01.394 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:01.640 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"full-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.64.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:54:01.641 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-65 2020-07-23T12:54:01.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:01.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:01.642 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:01.693 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:01.694 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:01.946 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.64.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:54:01.947 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-65 2020-07-23T12:54:01.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:01.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:01.949 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:01.996 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:01.996 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:02.308 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000011.64.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:54:02.309 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-65 2020-07-23T12:54:02.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:02.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:02.311 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:02.388 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:02.388 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:02.672 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.64.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:54:02.673 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-65 2020-07-23T12:54:02.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:02.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:02.675 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:02.719 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:02.719 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:02.986 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.64.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:54:02.988 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-65 2020-07-23T12:54:02.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:02.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:02.992 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:03.085 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:03.086 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:03.333 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.73.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:54:03.334 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-65 2020-07-23T12:54:03.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:03.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:03.336 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:03.379 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:03.379 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:03.679 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:54:03.680 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-65 2020-07-23T12:54:03.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:03.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:03.683 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:03.727 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:03.727 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:03.983 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.73.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:54:03.984 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-65 2020-07-23T12:54:03.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:03.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:03.986 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:04.035 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:04.036 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:04.296 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:54:04.296 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-65 2020-07-23T12:54:04.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:04.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:04.298 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:04.339 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:04.339 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:04.613 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.52.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:54:04.614 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-65 2020-07-23T12:54:04.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:04.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:04.615 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:04.699 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:04.700 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:04.984 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:54:04.985 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-65 2020-07-23T12:54:04.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:04.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:04.987 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:05.065 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:05.065 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:05.352 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.52.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:54:05.353 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-65 2020-07-23T12:54:05.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:05.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:05.355 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:05.399 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:05.399 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:05.645 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:54:05.646 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-65 2020-07-23T12:54:05.647 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-65 zones={test-zone2, test-zone1} 2020-07-23T12:54:06.856 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:54:06.864 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-07-23T12:54:06.865 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:54:06.914 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-07-23T12:54:06.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-07-23T12:54:07.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-07-23T12:54:07.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-07-23T12:54:08.095 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-65 zones={test-zone2, test-zone1} 2020-07-23T12:54:08.095 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-65 zones={test-zone3, test-zone1} 2020-07-23T12:54:08.730 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:54:08.778 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-07-23T12:54:08.779 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:54:08.788 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-07-23T12:54:08.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-07-23T12:54:09.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-07-23T12:54:09.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-07-23T12:54:10.019 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-65 zones={test-zone3, test-zone1} 2020-07-23T12:54:10.020 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-65 zones={test-zone3, test-zone2} 2020-07-23T12:54:10.135 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:54:10.144 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-07-23T12:54:10.145 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:54:10.194 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-07-23T12:54:10.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-07-23T12:54:10.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-07-23T12:54:10.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-07-23T12:54:11.318 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-65 zones={test-zone3, test-zone2} 2020-07-23T12:54:11.319 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... ok 2020-07-23T12:54:11.320 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-68 2020-07-23T12:54:11.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:11.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:11.941 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:11.979 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:11.979 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:12.222 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=a2b03b35-661e-4182-91b6-d60b3b65f694 2020-07-23T12:54:12.262 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595508851.841347_319.1","last_update":"2020-07-23T12:54:11.841347Z"}] 2020-07-23T12:54:12.263 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:54:12.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:12.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:12.265 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:12.305 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:12.306 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:12.560 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508838.229444_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:53:58.229444Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:54:12.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595508851.841347_319.1 target=1_1595508838.229444_315.1 2020-07-23T12:54:12.910 INFO:tasks.rgw.c2.client.1.smithi133.stdout:2020-07-23T12:54:12.905+0000 7f47447f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:54:12.911 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:54:12.905+0000 7f01aa7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:54:17.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:17.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:17.566 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:17.605 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:17.606 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:17.866 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508851.841347_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:11.841347Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:54:17.867 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:54:17.867 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:54:17.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:17.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:17.869 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:17.911 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:17.911 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:18.171 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508851.841347_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:11.841347Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:54:18.172 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:54:18.173 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:54:18.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:18.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:18.174 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:18.216 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:18.217 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:18.487 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508851.841347_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:11.841347Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:54:18.488 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:54:18.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:18.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:18.491 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-68 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:18.534 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:18.536 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:18.812 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.104.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.95.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-07-23T12:54:18.812 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-68 2020-07-23T12:54:18.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:18.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:18.814 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-68 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:18.860 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:18.860 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:19.134 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"testobj-sse-c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.104.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"testobj-sse-kms","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.95.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-07-23T12:54:19.134 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-68 2020-07-23T12:54:19.850 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... ok 2020-07-23T12:54:19.851 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-69 2020-07-23T12:54:20.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:20.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:20.188 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:20.225 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:20.226 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:20.473 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=a2b03b35-661e-4182-91b6-d60b3b65f694 2020-07-23T12:54:20.512 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595508859.990071_323.1","last_update":"2020-07-23T12:54:19.990071Z"}] 2020-07-23T12:54:20.513 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:54:20.514 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:20.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:20.515 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:20.561 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:20.561 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:20.813 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508851.841347_319.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:11.841347Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:54:20.814 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1595508859.990071_323.1 target=1_1595508851.841347_319.1 2020-07-23T12:54:21.120 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:54:21.116+0000 7fd09fff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:54:21.120 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:54:21.116+0000 7fd0a9ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:54:21.121 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:54:21.113+0000 7f01c17fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:54:21.122 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:54:21.116+0000 7fd09b7ee700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:54:21.122 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:54:21.113+0000 7f01a17fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:54:21.123 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:54:21.113+0000 7f01a97fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:54:21.123 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:54:21.117+0000 7f01c17fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:54:21.125 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:54:21.120+0000 7fd0a0ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:54:21.125 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:54:21.120+0000 7fd099feb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:54:21.126 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:54:21.120+0000 7fd09dff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:54:21.127 INFO:tasks.rgw.c1.client.1.smithi137.stdout:2020-07-23T12:54:21.124+0000 7fd0a3fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:54:21.128 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:54:21.121+0000 7f01aa7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:54:21.128 INFO:tasks.rgw.c2.client.0.smithi133.stdout:2020-07-23T12:54:21.121+0000 7f01997ea700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-07-23T12:54:25.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:25.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:25.818 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:25.859 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:25.859 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:26.120 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508859.990071_323.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:19.990071Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:54:26.121 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:54:26.121 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:54:26.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:26.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:26.123 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:26.165 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:26.165 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:26.434 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508859.990071_323.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:19.990071Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:54:26.435 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:54:26.436 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:54:26.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:26.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:26.439 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:26.480 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:26.481 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:26.758 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508859.990071_323.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:19.990071Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:54:26.758 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:54:26.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:26.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:26.762 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:26.804 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:26.805 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:27.079 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.68.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.68.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.77.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.78.5"}]} 2020-07-23T12:54:27.080 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-69 2020-07-23T12:54:27.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:27.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:27.082 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:27.127 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:27.128 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:27.399 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.77.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5"}}}] 2020-07-23T12:54:27.400 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-69 2020-07-23T12:54:27.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:27.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:27.402 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:27.444 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:27.444 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:27.719 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.68.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.68.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.77.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.78.5"}]} 2020-07-23T12:54:27.720 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-69 2020-07-23T12:54:27.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:27.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:27.722 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:27.804 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:27.805 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:28.095 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.77.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5"}}}] 2020-07-23T12:54:28.231 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-69 2020-07-23T12:54:28.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:28.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:28.235 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:28.285 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:28.285 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:28.576 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.68.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.68.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.77.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.78.5"}]} 2020-07-23T12:54:28.577 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-69 2020-07-23T12:54:28.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:28.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:28.578 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:28.626 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:28.626 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:28.890 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.68.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.68.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.77.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.78.5"}}}] 2020-07-23T12:54:28.891 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-69 2020-07-23T12:54:28.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:28.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:28.893 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:28.932 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:28.933 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:29.199 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.68.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.81.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.77.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.64.5"}]} 2020-07-23T12:54:29.200 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-69 2020-07-23T12:54:29.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:29.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:29.204 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:29.248 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:29.248 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:29.552 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.81.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.77.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}}] 2020-07-23T12:54:29.552 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-69 2020-07-23T12:54:29.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:29.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:29.555 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:29.597 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:29.598 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:29.857 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.68.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.81.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.77.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.64.5"}]} 2020-07-23T12:54:29.858 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-69 2020-07-23T12:54:29.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:29.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:29.860 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:29.902 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:29.903 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:30.175 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.81.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.77.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}}] 2020-07-23T12:54:30.176 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-69 2020-07-23T12:54:30.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:30.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:30.178 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:30.259 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:30.259 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:30.556 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.71.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.81.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.76.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.73.5"}]} 2020-07-23T12:54:30.557 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-69 2020-07-23T12:54:30.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:30.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:30.559 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:30.640 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:30.640 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:30.928 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.81.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}}] 2020-07-23T12:54:30.929 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-69 2020-07-23T12:54:30.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:30.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:30.931 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:31.015 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:31.016 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:31.254 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.71.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.81.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.76.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.73.5"}]} 2020-07-23T12:54:31.255 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-69 2020-07-23T12:54:31.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:31.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:31.257 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:31.301 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:31.302 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:31.585 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.81.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}}] 2020-07-23T12:54:31.586 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-69 2020-07-23T12:54:31.586 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-69 zones={test-zone2, test-zone1} 2020-07-23T12:54:32.607 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:54:32.616 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-07-23T12:54:32.617 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-07-23T12:54:32.618 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-07-23T12:54:32.618 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-07-23T12:54:32.619 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:54:32.628 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-07-23T12:54:32.629 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-07-23T12:54:32.629 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-07-23T12:54:32.630 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-07-23T12:54:32.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:32.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:32.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:32.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:32.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:33.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:33.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:33.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:33.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:33.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:33.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:33.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:33.787 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-69 zones={test-zone2, test-zone1} 2020-07-23T12:54:33.788 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-69 zones={test-zone3, test-zone1} 2020-07-23T12:54:34.279 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:54:34.291 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-07-23T12:54:34.291 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-07-23T12:54:34.292 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-07-23T12:54:34.292 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-07-23T12:54:34.293 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:54:34.299 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-07-23T12:54:34.299 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-07-23T12:54:34.300 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-07-23T12:54:34.300 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-07-23T12:54:34.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:34.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:34.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:34.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:34.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:34.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:34.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:34.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:35.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:35.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:35.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:35.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:35.439 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-69 zones={test-zone3, test-zone1} 2020-07-23T12:54:35.439 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-69 zones={test-zone3, test-zone2} 2020-07-23T12:54:36.027 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:54:36.046 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-07-23T12:54:36.046 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-07-23T12:54:36.047 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-07-23T12:54:36.047 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-07-23T12:54:36.048 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:54:36.055 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-07-23T12:54:36.056 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-07-23T12:54:36.056 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-07-23T12:54:36.057 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-07-23T12:54:36.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:36.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:36.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:36.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:36.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:36.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:36.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:36.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:36.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:36.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:36.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:37.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:37.193 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-69 zones={test-zone3, test-zone2} 2020-07-23T12:54:37.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:54:37.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:54:37.195 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:54:37.233 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:37.234 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:37.509 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=pdivcf-70 2020-07-23T12:54:38.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:38.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:38.255 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:38.297 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:38.297 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:38.529 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=a2b03b35-661e-4182-91b6-d60b3b65f694 2020-07-23T12:54:38.568 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595508878.056614_333.1","last_update":"2020-07-23T12:54:38.056614Z"}] 2020-07-23T12:54:38.569 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:54:38.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:38.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:38.571 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:38.609 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:38.609 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:38.866 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508878.056614_333.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:38.056614Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:54:38.867 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:54:38.867 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:54:38.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:38.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:38.870 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:38.914 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:38.914 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:39.177 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508878.056614_333.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:38.056614Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:54:39.177 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:54:39.178 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:54:39.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:39.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:39.180 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:39.218 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:39.219 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:39.582 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508878.056614_333.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:38.056614Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:54:39.583 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:54:39.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:39.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:39.586 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:39.635 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:39.636 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:39.903 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.73.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.55.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.80.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.81.5"}]} 2020-07-23T12:54:39.903 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-70 2020-07-23T12:54:39.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:39.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:39.906 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:39.950 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:39.950 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:40.231 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"full-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.80.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.81.5"}}}] 2020-07-23T12:54:40.232 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=pdivcf-70 2020-07-23T12:54:40.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:40.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:40.234 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:40.274 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:40.275 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:40.516 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.73.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.55.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.80.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.81.5"}]} 2020-07-23T12:54:40.517 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-70 2020-07-23T12:54:40.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:40.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:40.519 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:40.602 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:40.602 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:40.870 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.80.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.81.5"}}}] 2020-07-23T12:54:40.871 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=pdivcf-70 2020-07-23T12:54:40.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:40.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:40.873 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:40.912 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:40.912 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:41.176 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.73.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.55.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.80.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.81.5"}]} 2020-07-23T12:54:41.177 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-70 2020-07-23T12:54:41.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:41.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'pdivcf-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:41.179 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket pdivcf-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:41.222 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:41.223 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:41.500 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.73.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.55.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.80.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.81.5"}}}] 2020-07-23T12:54:41.501 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=pdivcf-70 2020-07-23T12:54:41.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:41.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:41.503 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:41.578 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:41.579 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:41.809 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.75.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.61.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.80.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.67.5"}]} 2020-07-23T12:54:41.809 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-70 2020-07-23T12:54:41.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:41.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:41.812 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:41.857 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:41.858 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:42.149 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.80.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}}] 2020-07-23T12:54:42.150 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=pdivcf-70 2020-07-23T12:54:42.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:42.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:42.152 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:42.193 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:42.194 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:42.441 INFO:teuthology.orchestra.run.smithi133.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.75.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.61.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.80.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.67.5"}]} 2020-07-23T12:54:42.442 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-70 2020-07-23T12:54:42.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:42.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:42.444 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket pdivcf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:42.495 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:42.496 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:42.777 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.80.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}}] 2020-07-23T12:54:42.778 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=pdivcf-70 2020-07-23T12:54:42.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:42.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:42.780 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:42.859 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:42.859 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:43.136 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.79.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.54.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.79.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.76.5"}]} 2020-07-23T12:54:43.137 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-70 2020-07-23T12:54:43.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:43.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:43.139 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:43.220 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:43.220 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:43.509 INFO:teuthology.orchestra.run.smithi137.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.79.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.79.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5"}}}] 2020-07-23T12:54:43.519 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=pdivcf-70 2020-07-23T12:54:43.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:43.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:43.521 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket pdivcf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:43.596 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:43.596 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:43.867 INFO:teuthology.orchestra.run.smithi137.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.79.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.54.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.79.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.76.5"}]} 2020-07-23T12:54:43.867 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-70 2020-07-23T12:54:43.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:43.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'pdivcf-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:43.870 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket pdivcf-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:43.915 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:43.916 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:44.194 INFO:teuthology.orchestra.run.smithi133.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.79.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.79.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5"}}}] 2020-07-23T12:54:44.195 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=pdivcf-70 2020-07-23T12:54:44.196 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-70 zones={test-zone2, test-zone1} 2020-07-23T12:54:45.141 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:54:45.148 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-07-23T12:54:45.149 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-07-23T12:54:45.149 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-07-23T12:54:45.150 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-07-23T12:54:45.150 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:54:45.160 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-07-23T12:54:45.161 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-07-23T12:54:45.161 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-07-23T12:54:45.162 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-07-23T12:54:45.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:45.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:45.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:45.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:45.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:45.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:45.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:45.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:45.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:46.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:46.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:46.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:46.342 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-70 zones={test-zone2, test-zone1} 2020-07-23T12:54:46.342 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-70 zones={test-zone3, test-zone1} 2020-07-23T12:54:47.050 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:54:47.057 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-07-23T12:54:47.058 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-07-23T12:54:47.058 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-07-23T12:54:47.059 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-07-23T12:54:47.059 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:54:47.066 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-07-23T12:54:47.067 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-07-23T12:54:47.067 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-07-23T12:54:47.067 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-07-23T12:54:47.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:47.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:47.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:47.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:47.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:47.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:47.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:47.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:47.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:47.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:47.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:48.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:48.199 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-70 zones={test-zone3, test-zone1} 2020-07-23T12:54:48.200 INFO:tasks.rgw_multi.zone_rados:comparing bucket=pdivcf-70 zones={test-zone3, test-zone2} 2020-07-23T12:54:48.518 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-07-23T12:54:48.528 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-07-23T12:54:48.529 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-07-23T12:54:48.530 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-07-23T12:54:48.531 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-07-23T12:54:48.532 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-07-23T12:54:48.539 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-07-23T12:54:48.540 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-07-23T12:54:48.541 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-07-23T12:54:48.541 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-07-23T12:54:48.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:48.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:48.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-07-23T12:54:48.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:48.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:49.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-07-23T12:54:49.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:49.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:49.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-07-23T12:54:49.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:49.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:49.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-07-23T12:54:49.698 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=pdivcf-70 zones={test-zone3, test-zone2} 2020-07-23T12:54:49.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:54:49.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:54:49.700 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:54:49.738 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:49.738 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:50.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:50.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'pdivcf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:50.049 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket pdivcf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:50.126 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:50.126 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:50.393 INFO:teuthology.orchestra.run.smithi137.stdout:[] 2020-07-23T12:54:50.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:50.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:50.396 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket pdivcf-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:50.476 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:50.476 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:50.746 INFO:teuthology.orchestra.run.smithi137.stdout:[{"op_id":"1#00000000001.68.5","op_tag":"0378667f-02f5-4ce6-824e-e2a396c12fc3.7551.4773","op":"write","object":"a","instance":"","state":"complete","index_ver":1,"timestamp":"2020-07-23T12:54:20.001463000Z","ver":{"pool":9,"epoch":5},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-69:0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.20"}]},{"op_id":"5#00000000001.68.5","op_tag":"0378667f-02f5-4ce6-824e-e2a396c12fc3.7551.4779","op":"write","object":"d","instance":"","state":"complete","index_ver":1,"timestamp":"2020-07-23T12:54:20.145432912Z","ver":{"pool":9,"epoch":3},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-69:0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.20"}]},{"op_id":"8#00000000001.77.5","op_tag":"0378667f-02f5-4ce6-824e-e2a396c12fc3.7551.4777","op":"write","object":"c","instance":"","state":"complete","index_ver":1,"timestamp":"2020-07-23T12:54:20.099062461Z","ver":{"pool":9,"epoch":6},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-69:0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.20"}]},{"op_id":"10#00000000001.78.5","op_tag":"0378667f-02f5-4ce6-824e-e2a396c12fc3.7551.4775","op":"write","object":"b","instance":"","state":"complete","index_ver":1,"timestamp":"2020-07-23T12:54:20.049612768Z","ver":{"pool":9,"epoch":5},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"0378667f-02f5-4ce6-824e-e2a396c12fc3:pdivcf-69:0378667f-02f5-4ce6-824e-e2a396c12fc3.7559.20"}]}] 2020-07-23T12:54:50.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:54:50.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-07-23T12:54:50.748 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-07-23T12:54:50.827 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:50.828 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:52.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:52.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'pdivcf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:52.456 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket pdivcf-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:52.535 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:52.536 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:52.835 INFO:teuthology.orchestra.run.smithi137.stdout:[] 2020-07-23T12:54:52.836 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... ok 2020-07-23T12:54:54.974 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... ok 2020-07-23T12:54:54.975 INFO:tasks.rgw_multisite_tests: 2020-07-23T12:54:54.977 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-07-23T12:54:54.977 INFO:tasks.rgw_multisite_tests:Ran 28 tests in 1401.099s 2020-07-23T12:54:54.978 INFO:tasks.rgw_multisite_tests: 2020-07-23T12:54:54.978 INFO:tasks.rgw_multisite_tests:OK 2020-07-23T12:54:54.982 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2020-07-23T12:54:54.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:54.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:54.988 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:55.026 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:55.026 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:55.291 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=a2b03b35-661e-4182-91b6-d60b3b65f694 2020-07-23T12:54:55.329 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595508878.056614_333.1","last_update":"2020-07-23T12:54:38.056614Z"}] 2020-07-23T12:54:55.330 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:54:55.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:55.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:55.331 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:55.372 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:55.372 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:55.633 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508878.056614_333.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:38.056614Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:54:55.634 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:54:55.635 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:54:55.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:55.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:55.638 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:55.683 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:55.683 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:55.975 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508878.056614_333.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:38.056614Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:54:55.976 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:54:55.977 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:54:55.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:55.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:55.979 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:56.022 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:56.023 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:56.296 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508878.056614_333.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:38.056614Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:54:56.297 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:54:56.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:56.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:56.300 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:56.340 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:56.341 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:56.584 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=a2b03b35-661e-4182-91b6-d60b3b65f694 2020-07-23T12:54:56.627 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595508878.056614_333.1","last_update":"2020-07-23T12:54:38.056614Z"}] 2020-07-23T12:54:56.628 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:54:56.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:56.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:56.631 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:56.670 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:56.671 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:56.948 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508878.056614_333.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:38.056614Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:54:56.948 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:54:57.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:57.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:57.479 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:57.515 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:57.515 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:57.775 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=a2b03b35-661e-4182-91b6-d60b3b65f694 2020-07-23T12:54:57.817 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595508897.472263_337.1","last_update":"2020-07-23T12:54:57.472263Z"}] 2020-07-23T12:54:57.817 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:54:57.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:57.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:57.824 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:57.864 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:57.865 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:58.131 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508897.472263_337.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:57.472263Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:54:58.131 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:54:58.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:58.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:58.373 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:58.413 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:58.413 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:58.678 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=a2b03b35-661e-4182-91b6-d60b3b65f694 2020-07-23T12:54:58.719 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595508897.472263_337.1","last_update":"2020-07-23T12:54:57.472263Z"}] 2020-07-23T12:54:58.720 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:54:58.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:58.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:58.721 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:58.760 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:58.761 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:59.030 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508897.472263_337.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:57.472263Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:54:59.030 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:54:59.569 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... ok 2020-07-23T12:54:59.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:59.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:59.571 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:59.608 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:59.609 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:54:59.871 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=a2b03b35-661e-4182-91b6-d60b3b65f694 2020-07-23T12:54:59.908 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595508899.564139_339.1","last_update":"2020-07-23T12:54:59.564139Z"}] 2020-07-23T12:54:59.908 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-07-23T12:54:59.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:59.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:54:59.910 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:54:59.948 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:54:59.949 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:55:00.183 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508899.564139_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:59.564139Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:55:00.183 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-07-23T12:55:00.184 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-07-23T12:55:00.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:55:00.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:55:00.188 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:55:00.228 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:55:00.229 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:55:00.517 INFO:teuthology.orchestra.run.smithi137.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508899.564139_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:59.564139Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:55:00.518 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-07-23T12:55:00.519 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:55:00.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:55:00.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:55:00.520 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:55:00.563 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:55:00.563 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:55:00.835 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508899.564139_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:59.564139Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:55:00.836 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:55:00.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:55:00.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:55:00.839 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:55:00.880 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:55:00.880 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:55:01.249 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=a2b03b35-661e-4182-91b6-d60b3b65f694 2020-07-23T12:55:01.288 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595508899.564139_339.1","last_update":"2020-07-23T12:54:59.564139Z"}] 2020-07-23T12:55:01.289 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:55:01.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:55:01.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:55:01.291 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:55:01.330 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:55:01.331 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:55:01.602 INFO:teuthology.orchestra.run.smithi133.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"a2b03b35-661e-4182-91b6-d60b3b65f694","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1595507375.329399_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-07-23T12:29:35.329399Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1595507528.176421_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:32:08.176421Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1595508899.564139_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-07-23T12:54:59.564139Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2020-07-23T12:55:01.603 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-07-23T12:55:01.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:55:01.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:55:01.808 INFO:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:55:01.843 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:55:01.844 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2020-07-23T12:55:02.095 INFO:teuthology.orchestra.run.smithi137.stderr:No --period given, using current period=a2b03b35-661e-4182-91b6-d60b3b65f694 2020-07-23T12:55:02.138 INFO:teuthology.orchestra.run.smithi137.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1595508901.801766_343.1","last_update":"2020-07-23T12:55:01.801766Z"}] 2020-07-23T12:55:02.139 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-07-23T12:55:02.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:55:02.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-07-23T12:55:02.141 INFO:teuthology.orchestra.run.smithi133:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-07-23T12:55:02.186 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setuser ceph since I am not root 2020-07-23T12:55:02.187 INFO:teuthology.orchestra.run.smithi133.stderr:ignoring --setgroup ceph since I am not root 2020-07-24T00:13:26.282 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-07-24T00:13:26.302 DEBUG:teuthology.task.console_log:Killing console logger for smithi137 2020-07-24T00:13:26.304 DEBUG:teuthology.task.console_log:Killing console logger for smithi133 2020-07-24T00:13:26.305 DEBUG:teuthology.task.console_log:Killing console logger for smithi137 2020-07-24T00:13:26.305 DEBUG:teuthology.task.console_log:Killing console logger for smithi133 2020-07-24T00:13:26.306 DEBUG:teuthology.exit:Finished running handlers